diff --git a/F2.qmd b/F2.qmd index a11e520..fc38c7d 100644 --- a/F2.qmd +++ b/F2.qmd @@ -5,11 +5,7 @@ Now that you know how images are viewed and what kinds of images exist in Earth # Image Manipulation: Bands, Arithmetic, Thresholds, and Masks - - - - -::: {.callout-tip} +:::{.callout-tip} # Chapter Information ## Author {.unlisted .unnumbered} @@ -35,46 +31,45 @@ Once images have been identified in Earth Engine, they can be viewed in a wide a * Import images and image collections, filter, and visualize (Part F1). -::: +::: ## Introduction {.unlisted .unnumbered} -Spectral indices are based on the fact that different objects and land covers on the Earth’s surface reflect different amounts of light from the Sun at different wavelengths. In the visible part of the spectrum, for example, a healthy green plant reflects a large amount of green light while absorbing blue and red light—which is why it appears green to our eyes. Light also arrives from the Sun at wavelengths outside what the human eye can see, and there are large differences in reflectances between living and nonliving land covers, and between different types of vegetation, both in the visible and outside the visible wavelengths. We visualized this earlier, in Chaps. F1.1 and F1.3 when we mapped color-infrared images (Fig. F2.0.1). +Spectral indices are based on the fact that different objects and land covers on the Earth’s surface reflect different amounts of light from the Sun at different wavelengths. In the visible part of the spectrum, for example, a healthy green plant reflects a large amount of green light while absorbing blue and red light—which is why it appears green to our eyes. Light also arrives from the Sun at wavelengths outside what the human eye can see, and there are large differences in reflectances between living and nonliving land covers, and between different types of vegetation, both in the visible and outside the visible wavelengths. We visualized this earlier, in Chaps. F1.1 and F1.3 when we mapped color-infrared images (Fig. F2.0.1). -![](F2/image39.png) +![Fig. F2.0.1 Mapped color-IR images from multiple satellite sensors that we mapped in Chap. F1.3. The near infrared spectrum is mapped as red, showing where there are high amounts of healthy vegetation.](F2/image39.png) -Fig. F2.0.1 Mapped color-IR images from multiple satellite sensors that we mapped in Chap. F1.3. The near infrared spectrum is mapped as red, showing where there are high amounts of healthy vegetation. If we graph the amount of light (reflectance) at different wavelengths that an object or land cover reflects, we can visualize this more easily (Fig. F2.0.2). For example, look at the reflectance curves for soil and water in the graph below. Soil and water both have relatively low reflectance at wavelengths around 300 nm (ultraviolet and violet light). Conversely, at wavelengths above 700 nm (red and infrared light) soil has relatively high reflectance, while water has very low reflectance. Vegetation, meanwhile, generally reflects large amounts of near infrared light, relative to other land covers. -![](F2/image32.png) +![Fig. F2.0.2 A graph of the amount of reflectance for different objects on the Earth’s surface at different wavelengths in the visible and infrared portions of the electromagnetic spectrum. 1 micrometer (µm) = 1,000 nanometers (nm).](F2/image32.png) -Fig. F2.0.2 A graph of the amount of reflectance for different objects on the Earth’s surface at different wavelengths in the visible and infrared portions of the electromagnetic spectrum. 1 micrometer (µm) = 1,000 nanometers (nm). -Spectral indices use math to express how objects reflect light across multiple portions of the spectrum as a single number. Indices combine multiple bands, often with simple operations of subtraction and division, to create a single value across an image that is intended to help to distinguish particular land uses or land covers of interest. Using Fig. F2.0.2, you can imagine which wavelengths might be the most informative for distinguishing among a variety of land covers. We will explore a variety of calculations made from combinations of bands in the following sections. +Spectral indices use math to express how objects reflect light across multiple portions of the spectrum as a single number. Indices combine multiple bands, often with simple operations of subtraction and division, to create a single value across an image that is intended to help to distinguish particular land uses or land covers of interest. Using Fig. F2.0.2, you can imagine which wavelengths might be the most informative for distinguishing among a variety of land covers. We will explore a variety of calculations made from combinations of bands in the following sections. Indices derived from satellite imagery are used as the basis of many remote-sensing analyses. Indices have been used in thousands of applications, from detecting anthropogenic deforestation to examining crop health. For example, the growth of economically important crops such as wheat and cotton can be monitored throughout the growing season: Bare soil reflects more red wavelengths, whereas growing crops reflect more of the near-infrared (NIR) wavelengths. Thus, calculating a ratio of these two bands can help monitor how well crops are growing (Jackson and Huete 1991). ## Band Arithmetic in Earth Engine -If you have not already done so, be sure to add the book’s code repository to the Code Editor by entering [](https://www.google.com/url?q=https://code.earthengine.google.com/?accept_repo%3Dprojects/gee-edu/book&sa=D&source=editors&ust=1671458829783542&usg=AOvVaw2f8xfEZP6c0zP_Ke8jL26U)[https://code.earthengine.google.com/?accept_repo=projects/gee-edu/book](https://www.google.com/url?q=https://code.earthengine.google.com/?accept_repo%3Dprojects/gee-edu/book&sa=D&source=editors&ust=1671458829783919&usg=AOvVaw2i09J44MzpMZkjV_JLEnNR) into your browser. The book’s scripts will then be available in the script manager panel. If you have trouble finding the repo, you can visit [this link](https://www.google.com/url?q=https://docs.google.com/presentation/d/1Kt6wGNoesYm__Cu3k3bnlbbyPN6m9SF4hQHK-pIDHfc/edit%23slide%3Did.g18a7b4b055d_0_624&sa=D&source=editors&ust=1671458829784270&usg=AOvVaw1Kr82KG60ZeFLYC8cOZ67A) for help. +If you have not already done so, be sure to add the book’s code repository to the Code Editor by entering [](https://www.google.com/url?q=https://code.earthengine.google.com/?accept_repo%3Dprojects/gee-edu/book&sa=D&source=editors&ust=1671458829783542&usg=AOvVaw2f8xfEZP6c0zP_Ke8jL26U)[https://code.earthengine.google.com/?accept_repo=projects/gee-edu/book](https://www.google.com/url?q=https://code.earthengine.google.com/?accept_repo%3Dprojects/gee-edu/book&sa=D&source=editors&ust=1671458829783919&usg=AOvVaw2i09J44MzpMZkjV_JLEnNR) into your browser. The book’s scripts will then be available in the script manager panel. If you have trouble finding the repo, you can visit [this link](https://www.google.com/url?q=https://docs.google.com/presentation/d/1Kt6wGNoesYm__Cu3k3bnlbbyPN6m9SF4hQHK-pIDHfc/edit%23slide%3Did.g18a7b4b055d_0_624&sa=D&source=editors&ust=1671458829784270&usg=AOvVaw1Kr82KG60ZeFLYC8cOZ67A) for help. Many indices can be calculated using band arithmetic in Earth Engine. Band arithmetic is the process of adding, subtracting, multiplying, or dividing two or more bands from an image. Here we’ll first do this manually, and then show you some more efficient ways to perform band arithmetic in Earth Engine. -### Arithmetic Calculation of NDVI +### Arithmetic Calculation of NDVI The red and near-infrared bands provide a lot of information about vegetation due to vegetation’s high reflectance in these wavelengths. Take a look at Fig. F2.0.2 and note, in particular, that vegetation curves (graphed in green) have relatively high reflectance in the NIR range (approximately 750–900 nm). Also note that vegetation has low reflectance in the red range (approximately 630–690 nm), where sunlight is absorbed by chlorophyll. This suggests that if the red and near-infrared bands could be combined, they would provide substantial information about vegetation. Soon after the launch of Landsat 1 in 1972, analysts worked to devise a robust single value that would convey the health of vegetation along a scale of −1 to 1. This yielded the NDVI, using the formula: -![](F2/image1.png)           (F2.0.1) +![](F2/image1.png) (F2.0.1) -where NIR and red refer to the brightness of each of those two bands. As seen in Chaps. F1.1 and F1.2, this brightness might be conveyed in units of reflectance, radiance, or digital number (DN); the NDVI is intended to give nearly equivalent values across platforms that use these wavelengths. The general form of this equation is called a “normalized difference”—the numerator is the “difference” and the denominator “normalizes” the value. Outputs for NDVI vary between −1 and 1. High amounts of green vegetation have values around 0.8–0.9. Absence of green leaves gives values near 0, and water gives values near −1. +where NIR and red refer to the brightness of each of those two bands. As seen in Chaps. F1.1 and F1.2, this brightness might be conveyed in units of reflectance, radiance, or digital number (DN); the NDVI is intended to give nearly equivalent values across platforms that use these wavelengths. The general form of this equation is called a “normalized difference”—the numerator is the “difference” and the denominator “normalizes” the value. Outputs for NDVI vary between −1 and 1. High amounts of green vegetation have values around 0.8–0.9. Absence of green leaves gives values near 0, and water gives values near −1. To compute the NDVI, we will introduce Earth Engine’s implementation of band arithmetic. Cloud-based band arithmetic is one of the most powerful aspects of Earth Engine, because the platform’s computers are optimized for this type of heavy processing. Arithmetic on bands can be done even at planetary scale very quickly—an idea that was out of reach before the advent of cloud-based remote sensing. Earth Engine automatically partitions calculations across a large number of computers as needed, and assembles the answer for display. As an example, let’s examine an image of San Francisco (​​Fig. F2.0.3). +```js ///// // Band Arithmetic ///// @@ -82,161 +77,168 @@ As an example, let’s examine an image of San Francisco (​​Fig. F2.0.3). // Calculate NDVI using Sentinel 2 // Import and filter imagery by location and date. -var sfoPoint = ee.Geometry.Point(-122.3774, 37.6194); -var sfoImage = ee.ImageCollection('COPERNICUS/S2') -   .filterBounds(sfoPoint) -   .filterDate('2020-02-01', '2020-04-01') -   .first(); +var sfoPoint = ee.Geometry.Point(-122.3774, 37.6194); +var sfoImage = ee.ImageCollection('COPERNICUS/S2') + .filterBounds(sfoPoint) + .filterDate('2020-02-01', '2020-04-01') + .first(); // Display the image as a false color composite. Map.centerObject(sfoImage, 11); Map.addLayer(sfoImage, { -   bands: ['B8', 'B4', 'B3'], -   min: 0, -   max: 2000}, 'False color'); + bands: ['B8', 'B4', 'B3'], + min: 0, + max: 2000}, 'False color'); -![](F2/image46.png) +``` +![Fig. F2.0.3 False color Sentinel-2 imagery of San Francisco and surroundings](F2/image46.png) -Fig. F2.0.3 False color Sentinel-2 imagery of San Francisco and surroundings -The simplest mathematical operations in Earth Engine are the add, subtract, multiply, and divide methods. Let’s select the near-infrared and red bands and use these operations to calculate NDVI for our image. +The simplest mathematical operations in Earth Engine are the add, subtract, multiply, and divide methods. Let’s select the near-infrared and red bands and use these operations to calculate NDVI for our image. +```js // Extract the near infrared and red bands. -var nir = sfoImage.select('B8'); -var red = sfoImage.select('B4'); +var nir = sfoImage.select('B8'); +var red = sfoImage.select('B4'); // Calculate the numerator and the denominator using subtraction and addition respectively. -var numerator = nir.subtract(red); -var denominator = nir.add(red); +var numerator = nir.subtract(red); +var denominator = nir.add(red); // Now calculate NDVI. -var ndvi = numerator.divide(denominator); +var ndvi = numerator.divide(denominator); // Add the layer to our map with a palette. -var vegPalette = ['red', 'white', 'green']; +var vegPalette = ['red', 'white', 'green']; Map.addLayer(ndvi, { -   min: -1, -   max: 1, -   palette: vegPalette + min: -1, + max: 1, + palette: vegPalette }, 'NDVI Manual'); +``` Examine the resulting index, using the Inspector to pick out pixel values in areas of vegetation and non-vegetation if desired. -![](F2/image50.png) +![Fig. F2.0.4 NDVI calculated using Sentinel-2. Remember that outputs for NDVI vary between −1 and 1. High amounts of green vegetation have values around 0.8–0.9. Absence of green leaves gives values near 0, and water gives values near −1.](F2/image50.png) -Fig. F2.0.4 NDVI calculated using Sentinel-2. Remember that outputs for NDVI vary between −1 and 1. High amounts of green vegetation have values around 0.8–0.9. Absence of green leaves gives values near 0, and water gives values near −1. Using these simple arithmetic tools, you can build almost any index, or develop and visualize your own. Earth Engine allows you to quickly and easily calculate and display the index across a large area. ### Single-Operation Computation of Normalized Difference for NDVI -Normalized differences like NDVI are so common in remote sensing that Earth Engine provides the ability to do that particular sequence of subtraction, addition, and division in a single step, using the normalizedDifference method. This method takes an input image, along with bands you specify, and creates a normalized difference of those two bands. The NDVI computation previously created with band arithmetic can be replaced with one line of code: +Normalized differences like NDVI are so common in remote sensing that Earth Engine provides the ability to do that particular sequence of subtraction, addition, and division in a single step, using the normalizedDifference method. This method takes an input image, along with bands you specify, and creates a normalized difference of those two bands. The NDVI computation previously created with band arithmetic can be replaced with one line of code: -// Now use the built-in normalizedDifference function to achieve the same outcome. -var ndviND = sfoImage.normalizedDifference(['B8', 'B4']); +```js +// Now use the built-in normalizedDifference function to achieve the same outcome. +var ndviND = sfoImage.normalizedDifference(['B8', 'B4']); Map.addLayer(ndviND, { -   min: -1, -   max: 1, -   palette: vegPalette + min: -1, + max: 1, + palette: vegPalette }, 'NDVI normalizedDiff'); -Note that the order in which you provide the two bands to normalizedDifference is important. We use B8, the near-infrared band, as the first parameter, and the red band B4 as the second. If your two computations of NDVI do not look identical when drawn to the screen, check to make sure that the order you have for the NIR and red bands is correct. +``` +Note that the order in which you provide the two bands to normalizedDifference is important. We use B8, the near-infrared band, as the first parameter, and the red band B4 as the second. If your two computations of NDVI do not look identical when drawn to the screen, check to make sure that the order you have for the NIR and red bands is correct. ### Using Normalized Difference for NDWI -As mentioned, the normalized difference approach is used for many different indices. Let’s apply the same normalizedDifference method to another index. +As mentioned, the normalized difference approach is used for many different indices. Let’s apply the same normalizedDifference method to another index. -The Normalized Difference Water Index (NDWI) was developed by Gao (1996) as an index of vegetation water content. The index is sensitive to changes in the liquid content of vegetation canopies. This means that the index can be used, for example, to detect vegetation experiencing drought conditions or differentiate crop irrigation levels. In dry areas, crops that are irrigated can be differentiated from natural vegetation. It is also sometimes called the Normalized Difference Moisture Index (NDMI). NDWI is formulated as follows: +The Normalized Difference Water Index (NDWI) was developed by Gao (1996) as an index of vegetation water content. The index is sensitive to changes in the liquid content of vegetation canopies. This means that the index can be used, for example, to detect vegetation experiencing drought conditions or differentiate crop irrigation levels. In dry areas, crops that are irrigated can be differentiated from natural vegetation. It is also sometimes called the Normalized Difference Moisture Index (NDMI). NDWI is formulated as follows: -![](F2/image2.png)                                (F2.0.2) +![(F2.0.2)](F2/image2.png) where NIR is near-infrared, centered near 860 nm (0.86 μm), and SWIR is short-wave infrared, centered near 1,240 nm (1.24 μm). -Compute and display NDWI in Earth Engine using the normalizedDifference method. Remember that for Sentinel-2, B8 is the NIR band and B11 is the SWIR band (refer to Chaps. F1.1 and F1.3 to find information about imagery bands). +Compute and display NDWI in Earth Engine using the normalizedDifference method. Remember that for Sentinel-2, B8 is the NIR band and B11 is the SWIR band (refer to Chaps. F1.1 and F1.3 to find information about imagery bands). +```js // Use normalizedDifference to calculate NDWI -var ndwi = sfoImage.normalizedDifference(['B8', 'B11']); -var waterPalette = ['white', 'blue']; +var ndwi = sfoImage.normalizedDifference(['B8', 'B11']); +var waterPalette = ['white', 'blue']; Map.addLayer(ndwi, { -   min: -0.5, -   max: 1, -   palette: waterPalette + min: -0.5, + max: 1, + palette: waterPalette }, 'NDWI'); +``` Examine the areas of the map that NDVI identified as having a lot of vegetation. Notice which are more blue. This is vegetation that has higher water content. -![](F2/image40.png) +![Fig. F2.0.5 NDWI displayed for Sentinel-2 over San Francisco](F2/image40.png) -Fig. F2.0.5 NDWI displayed for Sentinel-2 over San Francisco -::: {.callout-note} -Code Checkpoint F20a. The book’s repository contains a script that shows what your code should look like at this point.  +:::{.callout-note} +Code Checkpoint F20a. The book’s repository contains a script that shows what your code should look like at this point. ::: ## Thresholding, Masking, and Remapping Images -The previous section in this chapter discussed how to use band arithmetic to manipulate images. Those methods created new continuous values by combining bands within an image. This section uses logical operators to categorize band or index values to create a categorized image. +The previous section in this chapter discussed how to use band arithmetic to manipulate images. Those methods created new continuous values by combining bands within an image. This section uses logical operators to categorize band or index values to create a categorized image. ### Implementing a Threshold -Implementing a threshold uses a number (the threshold value) and logical operators to help us partition the variability of images into categories. For example, recall our map of NDVI. High amounts of vegetation have NDVI values near 1 and non-vegetated areas are near 0. If we want to see what areas of the map have vegetation, we can use a threshold to generalize the NDVI value in each pixel as being either “no vegetation” or “vegetation”. That is a substantial simplification, to be sure, but can help us to better comprehend the rich variation on the Earth’s surface. This type of categorization may be useful if, for example, we want to look at the proportion of a city that is vegetated. Let’s create a Sentinel-2 map of NDVI near Seattle, Washington, USA. Enter the code below in a new script. +Implementing a threshold uses a number (the threshold value) and logical operators to help us partition the variability of images into categories. For example, recall our map of NDVI. High amounts of vegetation have NDVI values near 1 and non-vegetated areas are near 0. If we want to see what areas of the map have vegetation, we can use a threshold to generalize the NDVI value in each pixel as being either “no vegetation” or “vegetation”. That is a substantial simplification, to be sure, but can help us to better comprehend the rich variation on the Earth’s surface. This type of categorization may be useful if, for example, we want to look at the proportion of a city that is vegetated. Let’s create a Sentinel-2 map of NDVI near Seattle, Washington, USA. Enter the code below in a new script. +```js // Create an NDVI image using Sentinel 2. -var seaPoint = ee.Geometry.Point(-122.2040, 47.6221); -var seaImage = ee.ImageCollection('COPERNICUS/S2') -   .filterBounds(seaPoint) -   .filterDate('2020-08-15', '2020-10-01') -   .first(); +var seaPoint = ee.Geometry.Point(-122.2040, 47.6221); +var seaImage = ee.ImageCollection('COPERNICUS/S2') + .filterBounds(seaPoint) + .filterDate('2020-08-15', '2020-10-01') + .first(); -var seaNDVI = seaImage.normalizedDifference(['B8', 'B4']); +var seaNDVI = seaImage.normalizedDifference(['B8', 'B4']); // And map it. Map.centerObject(seaPoint, 10); -var vegPalette = ['red', 'white', 'green']; +var vegPalette = ['red', 'white', 'green']; Map.addLayer(seaNDVI, -   { -       min: -1, -       max: 1, -       palette: vegPalette -   },   'NDVI Seattle'); + { + min: -1, + max: 1, + palette: vegPalette + }, 'NDVI Seattle'); -![](F2/image30.png) +``` +![Fig. F2.0.6 NDVI image of Sentinel-2 imagery over Seattle, Washington, USA](F2/image30.png) -Fig. F2.0.6 NDVI image of Sentinel-2 imagery over Seattle, Washington, USA -Inspect the image. We can see that vegetated areas are darker green while non-vegetated locations are white and water is pink. If we use the Inspector to query our image, we can see that parks and other forested areas have an NDVI over about 0.5. Thus, it would make sense to define areas with NDVI values greater than 0.5 as forested, and those below that threshold as not forested. +Inspect the image. We can see that vegetated areas are darker green while non-vegetated locations are white and water is pink. If we use the Inspector to query our image, we can see that parks and other forested areas have an NDVI over about 0.5. Thus, it would make sense to define areas with NDVI values greater than 0.5 as forested, and those below that threshold as not forested. Now let’s define that value as a threshold and use it to threshold our vegetated areas. +```js // Implement a threshold. -var seaVeg = seaNDVI.gt(0.5); +var seaVeg = seaNDVI.gt(0.5); // Map the threshold. Map.addLayer(seaVeg, -   { -       min: 0, -       max: 1, -       palette: ['white', 'green'] -   },   'Non-forest vs. Forest'); + { + min: 0, + max: 1, + palette: ['white', 'green'] + }, 'Non-forest vs. Forest'); -The gt method is from the family of Boolean operators—that is, gt is a function that performs a test in each pixel and returns the value 1 if the test evaluates to true, and 0 otherwise. Here, for every pixel in the image, it tests whether the NDVI value is greater than 0.5. When this condition is met, the layer seaVeg gets the value 1. When the condition is false, it receives the value 0. +``` +The gt method is from the family of Boolean operators—that is, gt is a function that performs a test in each pixel and returns the value 1 if the test evaluates to true, and 0 otherwise. Here, for every pixel in the image, it tests whether the NDVI value is greater than 0.5. When this condition is met, the layer seaVeg gets the value 1. When the condition is false, it receives the value 0. -![](F2/image47.png) +![Fig. F2.0.7 Thresholded forest and non-forest image based on NDVI for Seattle, Washington, USA](F2/image47.png) -Fig. F2.0.7 Thresholded forest and non-forest image based on NDVI for Seattle, Washington, USA -Use the Inspector tool to explore this new layer. If you click on a green location, that NDVI should be greater than 0.5. If you click on a white pixel, the NDVI value should be equal to or less than 0.5. +Use the Inspector tool to explore this new layer. If you click on a green location, that NDVI should be greater than 0.5. If you click on a white pixel, the NDVI value should be equal to or less than 0.5. Other operators in this Boolean family include less than (lt), less than or equal to (lte), equal to (eq), not equal to (neq), and greater than or equal to (gte) and more. -### Building Complex Categorizations with .where +### Building Complex Categorizations with .where -A binary map classifying NDVI is very useful. However, there are situations where you may want to split your image into more than two bins. Earth Engine provides a tool, the where method, that conditionally evaluates to true or false within each pixel depending on the outcome of a test. This is analogous to an if statement seen commonly in other languages. However, to perform this logic when programming for Earth Engine, we avoid using the JavaScript if statement. Importantly, JavaScript if commands are not calculated on Google’s servers, and can create serious problems when running your code—in effect, the servers try to ship all of the information to be executed to your own computer’s browser, which is very underequipped for such enormous tasks. Instead, we use the where clause for conditional logic. +A binary map classifying NDVI is very useful. However, there are situations where you may want to split your image into more than two bins. Earth Engine provides a tool, the where method, that conditionally evaluates to true or false within each pixel depending on the outcome of a test. This is analogous to an if statement seen commonly in other languages. However, to perform this logic when programming for Earth Engine, we avoid using the JavaScript if statement. Importantly, JavaScript if commands are not calculated on Google’s servers, and can create serious problems when running your code—in effect, the servers try to ship all of the information to be executed to your own computer’s browser, which is very underequipped for such enormous tasks. Instead, we use the where clause for conditional logic. -Suppose instead of just splitting the forested areas from the non-forested areas in our NDVI, we want to split the image into likely water, non-forested, and forested areas. We can use where and thresholds of -0.1 and 0.5. We will start by creating an image using ee.Image. We then clip the new image so that it covers the same area as our seaNDVI layer. +Suppose instead of just splitting the forested areas from the non-forested areas in our NDVI, we want to split the image into likely water, non-forested, and forested areas. We can use where and thresholds of -0.1 and 0.5. We will start by creating an image using ee.Image. We then clip the new image so that it covers the same area as our seaNDVI layer. +```js // Implement .where. // Create a starting image with all values = 1. -var seaWhere = ee.Image(1)   // Use clip to constrain the size of the new image.   .clip(seaNDVI.geometry()); +var seaWhere = ee.Image(1) // Use clip to constrain the size of the new image. .clip(seaNDVI.geometry()); // Make all NDVI values less than -0.1 equal 0. seaWhere = seaWhere.where(seaNDVI.lte(-0.1), 0); @@ -246,101 +248,107 @@ seaWhere = seaWhere.where(seaNDVI.gte(0.5), 2); // Map our layer that has been divided into three classes. Map.addLayer(seaWhere, -   { -       min: 0, -       max: 2, -       palette: ['blue', 'white', 'green'] -   },   'Water, Non-forest, Forest'); + { + min: 0, + max: 2, + palette: ['blue', 'white', 'green'] + }, 'Water, Non-forest, Forest'); -There are a few interesting things to note about this code that you may not have seen before. First, we’re not defining a new variable for each where call. As a result, we can perform many where calls without creating a new variable each time and needing to keep track of them. Second, when we created the starting image, we set the value to 1. This means that we could easily set the bottom and top values with one where clause each. Finally, while we did not do it here, we can combine multiple where clauses using and and or. For example, we could identify pixels with an intermediate level of NDVI using seaNDVI.gte(-0.1).and(seaNDVI.lt(0.5)). +``` +There are a few interesting things to note about this code that you may not have seen before. First, we’re not defining a new variable for each where call. As a result, we can perform many where calls without creating a new variable each time and needing to keep track of them. Second, when we created the starting image, we set the value to 1. This means that we could easily set the bottom and top values with one where clause each. Finally, while we did not do it here, we can combine multiple where clauses using and and or. For example, we could identify pixels with an intermediate level of NDVI using seaNDVI.gte(-0.1).and(seaNDVI.lt(0.5)). -![](F2/image37.png) +![Fig. F2.0.8 Thresholded water, forest, and non-forest image based on NDVI for Seattle, Washington, USA.](F2/image37.png) -Fig. F2.0.8 Thresholded water, forest, and non-forest image based on NDVI for Seattle, Washington, USA. ### Masking Specific Values in an Image Masking an image is a technique that removes specific areas of an image—those covered by the mask—from being displayed or analyzed. Earth Engine allows you to both view the current mask and update the mask. +```js // Implement masking. // View the seaVeg layer's current mask. Map.centerObject(seaPoint, 9); Map.addLayer(seaVeg.mask(), {}, 'seaVeg Mask'); -![](F2/image23.png) +``` +![Fig. F2.0.9 The existing mask for the seaVeg layer we created previously](F2/image23.png) -Fig. F2.0.9 The existing mask for the seaVeg layer we created previously You can use the Inspector to see that the black area is masked and the white area has a constant value of 1. This means that data values are mapped and available for analysis within the white area only. -Now suppose we only want to display and conduct analyses in the forested areas. Let’s mask out the non-forested areas from our image. First, we create a binary mask using the equals (eq) method. +Now suppose we only want to display and conduct analyses in the forested areas. Let’s mask out the non-forested areas from our image. First, we create a binary mask using the equals (eq) method. +```js // Create a binary mask of non-forest. -var vegMask = seaVeg.eq(1); +var vegMask = seaVeg.eq(1); -In making a mask, you set the values you want to see and analyze to be a number greater than 0. The idea is to set unwanted values to get the value of 0. Pixels that had 0 values become masked out (in practice, they do not appear on the screen at all) once we use the updateMask method to add these values to the existing mask. +``` +In making a mask, you set the values you want to see and analyze to be a number greater than 0. The idea is to set unwanted values to get the value of 0. Pixels that had 0 values become masked out (in practice, they do not appear on the screen at all) once we use the updateMask method to add these values to the existing mask. +```js // Update the seaVeg mask with the non-forest mask. -var maskedVeg = seaVeg.updateMask(vegMask); +var maskedVeg = seaVeg.updateMask(vegMask); // Map the updated Veg layer Map.addLayer(maskedVeg, -   { -       min: 0, -       max: 1, -       palette: ['green'] -   },   'Masked Forest Layer'); + { + min: 0, + max: 1, + palette: ['green'] + }, 'Masked Forest Layer'); -Turn off all of the other layers. You can see how the maskedVeg layer now has masked out all non-forested areas. +``` +Turn off all of the other layers. You can see how the maskedVeg layer now has masked out all non-forested areas. -![](F2/image26.png) +![Fig. F2.0.10 An updated mask now displays only the forested areas. Non-forested areas are masked out and transparent.](F2/image26.png) -Fig. F2.0.10 An updated mask now displays only the forested areas. Non-forested areas are masked out and transparent. Map the updated mask for the layer and you can see why this is. +```js // Map the updated mask Map.addLayer(maskedVeg.mask(), {}, 'maskedVeg Mask'); -![](F2/image33.png) +``` +![Fig. F2.0.11 The updated mask. Areas of non-forest are now masked out as well (black areas of the image).](F2/image33.png) -Fig. F2.0.11 The updated mask. Areas of non-forest are now masked out as well (black areas of the image). ### Remapping Values in an Image Remapping takes specific values in an image and assigns them a different value. This is particularly useful for categorical datasets, including those you read about in Chap. F1.2 and those we have created earlier in this chapter. -Let’s use the remap method to change the values for our seaWhere layer. Note that since we’re changing the middle value to be the largest, we’ll need to adjust our palette as well. +Let’s use the remap method to change the values for our seaWhere layer. Note that since we’re changing the middle value to be the largest, we’ll need to adjust our palette as well. +```js // Implement remapping. // Remap the values from the seaWhere layer. -var seaRemap = seaWhere.remap([0, 1, 2], // Existing values.   [9, 11, 10]); // Remapped values. +var seaRemap = seaWhere.remap([0, 1, 2], // Existing values. [9, 11, 10]); // Remapped values. Map.addLayer(seaRemap, -   { -       min: 9, -       max: 11, -       palette: ['blue', 'green', 'white'] -   },   'Remapped Values'); + { + min: 9, + max: 11, + palette: ['blue', 'green', 'white'] + }, 'Remapped Values'); -Use the inspector to compare values between our original seaWhere (displayed as Water, Non-Forest, Forest) and the seaRemap, marked as “Remapped Values.” Click on a forested area and you should see that the Remapped Values should be 10, instead of 2 (Fig. F2.0.12). +``` +Use the inspector to compare values between our original seaWhere (displayed as Water, Non-Forest, Forest) and the seaRemap, marked as “Remapped Values.” Click on a forested area and you should see that the Remapped Values should be 10, instead of 2 (Fig. F2.0.12). -![](F2/image28.png) +![Fig. F2.0.12 For forested areas, the remapped layer has a value of 10, compared with the original layer, which has a value of 2. You may have more layers in your Inspector.](F2/image28.png) -Fig. F2.0.12 For forested areas, the remapped layer has a value of 10, compared with the original layer, which has a value of 2. You may have more layers in your Inspector. -::: {.callout-note} -Code Checkpoint F20b. The book’s repository contains a script that shows what your code should look like at this point.  +:::{.callout-note} +Code Checkpoint F20b. The book’s repository contains a script that shows what your code should look like at this point. ::: ## Synthesis {.unnumbered} -Assignment 1. In addition to vegetation indices and other land cover indices, you can use properties of different soil types to create geological indices. The Clay Minerals Ratio (CMR) is one of these. This index highlights soils containing clay and alunite, which absorb radiation in the SWIR portion (2.0–2.3 μm) of the spectrum. +Assignment 1. In addition to vegetation indices and other land cover indices, you can use properties of different soil types to create geological indices. The Clay Minerals Ratio (CMR) is one of these. This index highlights soils containing clay and alunite, which absorb radiation in the SWIR portion (2.0–2.3 μm) of the spectrum. ![](F2/image3.png) -SWIR 1 should be in the 1.55–1.75 µm range, and SWIR 2 should be in the 2.08–2.35 µm range. Calculate and display CMR at the following point: ee.Geometry.Point(-100.543, 33.456). Don’t forget to use Map.centerObject. +SWIR 1 should be in the 1.55–1.75 µm range, and SWIR 2 should be in the 2.08–2.35 µm range. Calculate and display CMR at the following point: ee.Geometry.Point(-100.543, 33.456). Don’t forget to use Map.centerObject. -We’ve selected an area of Texas known for its clay soils. Compare this with an area without clay soils (for example, try an area around Seattle or Tacoma, Washington, USA). Note that this index will also pick up roads and other paved areas. +We’ve selected an area of Texas known for its clay soils. Compare this with an area without clay soils (for example, try an area around Seattle or Tacoma, Washington, USA). Note that this index will also pick up roads and other paved areas. Assignment 2. Calculate the Iron Oxide Ratio, which can be used to detect hydrothermally altered rocks (e.g., from volcanoes) that contain iron-bearing sulfides which have been oxidized (Segal, 1982). @@ -348,11 +356,11 @@ Here’s the formula: ![](F2/image4.png) -Red should be the 0.63–0.69 µm spectral range and Blue the 0.45–0.52 µm. Using Landsat 8, you can also find an interesting area to map by considering where these types of rocks might occur. +Red should be the 0.63–0.69 µm spectral range and Blue the 0.45–0.52 µm. Using Landsat 8, you can also find an interesting area to map by considering where these types of rocks might occur. -Assignment 3. Calculate the Normalized Difference Built-Up Index (NDBI) for the sfoImage used in this chapter. +Assignment 3. Calculate the Normalized Difference Built-Up Index (NDBI) for the sfoImage used in this chapter. -The NDBI was developed by Zha et al. (2003) to aid in differentiating urban areas (e.g., densely clustered buildings and roads) from other land cover types. The index exploits the fact that urban areas, which generally have a great deal of impervious surface cover, reflect SWIR very strongly. If you like, refer back to Fig. F2.0.2. +The NDBI was developed by Zha et al. (2003) to aid in differentiating urban areas (e.g., densely clustered buildings and roads) from other land cover types. The index exploits the fact that urban areas, which generally have a great deal of impervious surface cover, reflect SWIR very strongly. If you like, refer back to Fig. F2.0.2. The formula is: @@ -360,13 +368,13 @@ The formula is: Using what we know about Sentinel-2 bands, compute NDBI and display it. -Bonus: Note that NDBI is the negative of NDWI computed earlier. We can prove this by using the JavaScript reverse method to reverse the palette used for NDWI in Earth Engine. This method reverses the order of items in the JavaScript list. Create a new palette for NDBI using the reverse method and display the map. As a hint, here is code to use the reverse method. +Bonus: Note that NDBI is the negative of NDWI computed earlier. We can prove this by using the JavaScript reverse method to reverse the palette used for NDWI in Earth Engine. This method reverses the order of items in the JavaScript list. Create a new palette for NDBI using the reverse method and display the map. As a hint, here is code to use the reverse method. -var barePalette = waterPalette.reverse(); +var barePalette = waterPalette.reverse(); ## Conclusion {.unnumbered} -In this chapter, you learned how to select multiple bands from an image and calculate indices. You also learned about thresholding values in an image, slicing them into multiple categories using thresholds. It is also possible to work with one set of class numbers and remap them quickly to another set. Using these techniques, you have some of the basic tools of image manipulation. In subsequent chapters you will encounter more complex and specialized image manipulation techniques, including pixel-based image transformations (Chap. F3.1), neighborhood-based image transformations (Chap. F3.2), and object-based image analysis (Chap. F3.3). +In this chapter, you learned how to select multiple bands from an image and calculate indices. You also learned about thresholding values in an image, slicing them into multiple categories using thresholds. It is also possible to work with one set of class numbers and remap them quickly to another set. Using these techniques, you have some of the basic tools of image manipulation. In subsequent chapters you will encounter more complex and specialized image manipulation techniques, including pixel-based image transformations (Chap. F3.1), neighborhood-based image transformations (Chap. F3.2), and object-based image analysis (Chap. F3.3). ## References {.unnumbered} @@ -398,13 +406,13 @@ Souza Jr CM, Siqueira JV, Sales MH, et al (2013) Ten-year Landsat classification -# Interpreting an Image: Classification +# Interpreting an Image: Classification -::: {.callout-tip} +:::{.callout-tip} # Chapter Information ## Author {.unlisted .unnumbered} @@ -418,7 +426,7 @@ Andréa Puzzi Nicolau, Karen Dyson, David Saah, Nicholas Clinton ## Overview {.unlisted .unnumbered} -Image classification is a fundamental goal of remote sensing. It takes the user from viewing an image to labeling its contents. This chapter introduces readers to the concept of classification and walks users through the many options for image classification in Earth Engine. You will explore the processes of training data collection, classifier selection, classifier training, and image classification. +Image classification is a fundamental goal of remote sensing. It takes the user from viewing an image to labeling its contents. This chapter introduces readers to the concept of classification and walks users through the many options for image classification in Earth Engine. You will explore the processes of training data collection, classifier selection, classifier training, and image classification. ## Learning Outcomes {.unlisted .unnumbered} @@ -435,64 +443,64 @@ Image classification is a fundamental goal of remote sensing. It takes the user * Import images and image collections, filter, and visualize (Part F1). * Understand bands and how to select them (Chap. F1.2, Chap. F2.0). -::: +::: ## Introduction {.unlisted .unnumbered} -Classification is addressed in a broad range of fields, including mathematics, statistics, data mining, machine learning, and more. For a deeper treatment of classification, interested readers may see some of the following suggestions: Witten et al. (2011), Hastie et al. (2009), Goodfellow et al. (2016), Gareth et al. (2013), Géron (2019), Müller et al. (2016), or Witten et al. (2005). Unlike regression, which predicts continuous variables, classification predicts categorical, or discrete, variables—variables with a finite number of categories (e.g., age range). +Classification is addressed in a broad range of fields, including mathematics, statistics, data mining, machine learning, and more. For a deeper treatment of classification, interested readers may see some of the following suggestions: Witten et al. (2011), Hastie et al. (2009), Goodfellow et al. (2016), Gareth et al. (2013), Géron (2019), Müller et al. (2016), or Witten et al. (2005). Unlike regression, which predicts continuous variables, classification predicts categorical, or discrete, variables—variables with a finite number of categories (e.g., age range). -In remote sensing, image classification is an attempt to categorize all pixels in an image into a finite number of labeled land cover and/or land use classes. The resulting classified image is a simplified thematic map derived from the original image (Fig. F2.1.1). Land cover and land use information is essential for many environmental and socioeconomic applications, including natural resource management, urban planning, biodiversity conservation, agricultural monitoring, and carbon accounting. +In remote sensing, image classification is an attempt to categorize all pixels in an image into a finite number of labeled land cover and/or land use classes. The resulting classified image is a simplified thematic map derived from the original image (Fig. F2.1.1). Land cover and land use information is essential for many environmental and socioeconomic applications, including natural resource management, urban planning, biodiversity conservation, agricultural monitoring, and carbon accounting. -![](F2/image48.png) +![Fig. F2.1.1 Image classification concept](F2/image48.png) -Fig. F2.1.1 Image classification concept Image classification techniques for generating land cover and land use information have been in use since the 1980s (Li et al. 2014). Here, we will cover the concepts of pixel-based supervised and unsupervised classifications, testing out different classifiers. Chapter F3.3 covers the concept and application of object-based classification. -It is important to define land use and land cover. Land cover relates to the physical characteristics of the surface: simply put, it documents whether an area of the Earth’s surface is covered by forests, water, impervious surfaces, etc. Land use refers to how this land is being used by people. For example, herbaceous vegetation is considered a land cover but can indicate different land uses: the grass in a pasture is an agricultural land use, whereas the grass in an urban area can be classified as a park. +It is important to define land use and land cover. Land cover relates to the physical characteristics of the surface: simply put, it documents whether an area of the Earth’s surface is covered by forests, water, impervious surfaces, etc. Land use refers to how this land is being used by people. For example, herbaceous vegetation is considered a land cover but can indicate different land uses: the grass in a pasture is an agricultural land use, whereas the grass in an urban area can be classified as a park. ## Supervised Classification -If you have not already done so, be sure to add the book’s code repository to the Code Editor by entering [](https://www.google.com/url?q=https://code.earthengine.google.com/?accept_repo%3Dprojects/gee-edu/book&sa=D&source=editors&ust=1671458829866098&usg=AOvVaw16x5swm9HlorS5Mbw7E42X)[https://code.earthengine.google.com/?accept_repo=projects/gee-edu/book](https://www.google.com/url?q=https://code.earthengine.google.com/?accept_repo%3Dprojects/gee-edu/book&sa=D&source=editors&ust=1671458829866485&usg=AOvVaw0-N-JCWWgnM493BKa7Ichm) into your browser. The book’s scripts will then be available in the script manager panel. If you have trouble finding the repo, you can visit [this link](https://www.google.com/url?q=https://docs.google.com/presentation/d/1Kt6wGNoesYm__Cu3k3bnlbbyPN6m9SF4hQHK-pIDHfc/edit%23slide%3Did.g18a7b4b055d_0_624&sa=D&source=editors&ust=1671458829866823&usg=AOvVaw0ytMyRvutssBcVr2GdcBHA) for help. +If you have not already done so, be sure to add the book’s code repository to the Code Editor by entering [](https://www.google.com/url?q=https://code.earthengine.google.com/?accept_repo%3Dprojects/gee-edu/book&sa=D&source=editors&ust=1671458829866098&usg=AOvVaw16x5swm9HlorS5Mbw7E42X)[https://code.earthengine.google.com/?accept_repo=projects/gee-edu/book](https://www.google.com/url?q=https://code.earthengine.google.com/?accept_repo%3Dprojects/gee-edu/book&sa=D&source=editors&ust=1671458829866485&usg=AOvVaw0-N-JCWWgnM493BKa7Ichm) into your browser. The book’s scripts will then be available in the script manager panel. If you have trouble finding the repo, you can visit [this link](https://www.google.com/url?q=https://docs.google.com/presentation/d/1Kt6wGNoesYm__Cu3k3bnlbbyPN6m9SF4hQHK-pIDHfc/edit%23slide%3Did.g18a7b4b055d_0_624&sa=D&source=editors&ust=1671458829866823&usg=AOvVaw0ytMyRvutssBcVr2GdcBHA) for help. -Supervised classification uses a training dataset with known labels and representing the spectral characteristics of each land cover class of interest to “supervise” the classification. The overall approach of a supervised classification in Earth Engine is summarized as follows: +Supervised classification uses a training dataset with known labels and representing the spectral characteristics of each land cover class of interest to “supervise” the classification. The overall approach of a supervised classification in Earth Engine is summarized as follows: 1. Get a scene. 2. Collect training data. 3. Select and train a classifier using the training data. 4. Classify the image using the selected classifier. -We will begin by creating training data manually, based on a clear Landsat image (Fig. F2.1.2). Copy the code block below to define your Landsat 8 scene variable and add it to the map. We will use a point in Milan, Italy, as the center of the area for our image classification. +We will begin by creating training data manually, based on a clear Landsat image (Fig. F2.1.2). Copy the code block below to define your Landsat 8 scene variable and add it to the map. We will use a point in Milan, Italy, as the center of the area for our image classification. +```js // Create an Earth Engine Point object over Milan. -var pt = ee.Geometry.Point([9.453, 45.424]); +var pt = ee.Geometry.Point([9.453, 45.424]); // Filter the Landsat 8 collection and select the least cloudy image. -var landsat = ee.ImageCollection('LANDSAT/LC08/C02/T1_L2') -   .filterBounds(pt) -   .filterDate('2019-01-01', '2020-01-01') -   .sort('CLOUD_COVER') -   .first(); +var landsat = ee.ImageCollection('LANDSAT/LC08/C02/T1_L2') + .filterBounds(pt) + .filterDate('2019-01-01', '2020-01-01') + .sort('CLOUD_COVER') + .first(); // Center the map on that image. Map.centerObject(landsat, 8); // Add Landsat image to the map. -var visParams = { -   bands: ['SR_B4', 'SR_B3', 'SR_B2'], -   min: 7000, -   max: 12000 +var visParams = { + bands: ['SR_B4', 'SR_B3', 'SR_B2'], + min: 7000, + max: 12000 }; Map.addLayer(landsat, visParams, 'Landsat 8 image'); -![](F2/image44.png) +``` +![Fig. F2.1.2 Landsat image](F2/image44.png) -Fig. F2.1.2 Landsat image Using the Geometry Tools, we will create points on the Landsat image that represent land cover classes of interest to use as our training data. We’ll need to do two things: (1) identify where each land cover occurs on the ground, and (2) label the points with the proper class number. For this exercise, we will use the classes and codes shown in Table 2.1.1. -Table 2.1.1 Land cover classes +Table 2.1.1 Land cover classes Class @@ -514,169 +522,168 @@ Herbaceous 3 -In the Geometry Tools, click on the marker option (Fig. F2.1.3). This will create a point geometry which will show up as an import named “geometry”. Click on the gear icon to configure this import. +In the Geometry Tools, click on the marker option (Fig. F2.1.3). This will create a point geometry which will show up as an import named “geometry”. Click on the gear icon to configure this import. -![](F2/image22.png) +![Fig. F2.1.3 Creating a new layer in the Geometry Imports](F2/image22.png) -Fig. F2.1.3 Creating a new layer in the Geometry Imports We will start by collecting forest points, so name the import forest. Import it as a FeatureCollection, and then click + Property. Name the new property “class” and give it a value of 0 (Fig. F2.1.4). We can also choose a color to represent this class. For a forest class, it is natural to choose a green color. You can choose the color you prefer by clicking on it, or, for more control, you can use a hexadecimal value. -Hexadecimal values are used throughout the digital world to represent specific colors across computers and operating systems. They are specified by six values arranged in three pairs, with one pair each for the red, green, and blue brightness values. If you’re unfamiliar with hexadecimal values, imagine for a moment that colors were specified in pairs of base 10 numbers instead of pairs of base 16. In that case, a bright pure red value would be “990000”; a bright pure green value would be “009900”; and a bright pure blue value would be “000099”. A value like “501263” would be a mixture of the three colors, not especially bright, having roughly equal amounts of blue and red, and much less green: a color that would be a shade of purple. To create numbers in the hexadecimal system, which might feel entirely natural if humans had evolved to have 16 fingers, sixteen “digits” are needed: a base 16 counter goes 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, A, B, C, D, E, F, then 10, 11, and so on. Given that counting framework, the number “FF” is like “99” in base 10: the largest two-digit number. The hexadecimal color used for coloring the letters of the word FeatureCollection in this book, a color with roughly equal amounts of blue and red, and much less green, is “7F1FA2”   +Hexadecimal values are used throughout the digital world to represent specific colors across computers and operating systems. They are specified by six values arranged in three pairs, with one pair each for the red, green, and blue brightness values. If you’re unfamiliar with hexadecimal values, imagine for a moment that colors were specified in pairs of base 10 numbers instead of pairs of base 16. In that case, a bright pure red value would be “990000”; a bright pure green value would be “009900”; and a bright pure blue value would be “000099”. A value like “501263” would be a mixture of the three colors, not especially bright, having roughly equal amounts of blue and red, and much less green: a color that would be a shade of purple. To create numbers in the hexadecimal system, which might feel entirely natural if humans had evolved to have 16 fingers, sixteen “digits” are needed: a base 16 counter goes 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, A, B, C, D, E, F, then 10, 11, and so on. Given that counting framework, the number “FF” is like “99” in base 10: the largest two-digit number. The hexadecimal color used for coloring the letters of the word FeatureCollection in this book, a color with roughly equal amounts of blue and red, and much less green, is “7F1FA2” -Returning to the coloring of the forest points, the hexadecimal value “589400” is a little bit of red, about twice as much green, and no blue: the deep green seen in Figure F2.1.4. Enter that value, with or without the “#” in front, and click OK after finishing the configuration. +Returning to the coloring of the forest points, the hexadecimal value “589400” is a little bit of red, about twice as much green, and no blue: the deep green seen in Figure F2.1.4. Enter that value, with or without the “#” in front, and click OK after finishing the configuration. -![](F2/image36.png) +![Fig. F2.1.4 Edit geometry layer properties](F2/image36.png) -Fig. F2.1.4 Edit geometry layer properties -Now, in the Geometry Imports, we will see that the import has been renamed forest. Click on it to activate the drawing mode (Fig. F2.1.5) in order to start collecting forest points. +Now, in the Geometry Imports, we will see that the import has been renamed forest. Click on it to activate the drawing mode (Fig. F2.1.5) in order to start collecting forest points. -![](F2/image29.png) +![Fig. F2.1.5 Activate forest layer to start collection](F2/image29.png) -Fig. F2.1.5 Activate forest layer to start collection -Now, start collecting points over forested areas (Fig. F2.1.6). Zoom in and out as needed. You can use the satellite basemap to assist you, but the basis of your collection should be the Landsat image. Remember that the more points you collect, the more the classifier will learn from the information you provide. For now, let’s set a goal to collect 25 points per class. Click Exit next to Point drawing (Fig. F2.1.5) when finished. +Now, start collecting points over forested areas (Fig. F2.1.6). Zoom in and out as needed. You can use the satellite basemap to assist you, but the basis of your collection should be the Landsat image. Remember that the more points you collect, the more the classifier will learn from the information you provide. For now, let’s set a goal to collect 25 points per class. Click Exit next to Point drawing (Fig. F2.1.5) when finished. -![](F2/image38.png) +![Fig. F2.1.6 Forest points](F2/image38.png) -Fig. F2.1.6 Forest points -Repeat the same process for the other classes by creating new layers (Fig. F2.1.7). Don’t forget to import using the FeatureCollection option as mentioned above. For the developed class, collect points over urban areas. For the water class, collect points over the Ligurian Sea, and also look for other bodies of water, like rivers. For the herbaceous class, collect points over agricultural fields. Remember to set the “class” property for each class to its corresponding code (see Table 2.1.1) and click Exit once you finalize collecting points for each class as mentioned above. We will be using the following hexadecimal colors for the other classes: #FF0000 for developed, #1A11FF for water, and #D0741E for herbaceous.  +Repeat the same process for the other classes by creating new layers (Fig. F2.1.7). Don’t forget to import using the FeatureCollection option as mentioned above. For the developed class, collect points over urban areas. For the water class, collect points over the Ligurian Sea, and also look for other bodies of water, like rivers. For the herbaceous class, collect points over agricultural fields. Remember to set the “class” property for each class to its corresponding code (see Table 2.1.1) and click Exit once you finalize collecting points for each class as mentioned above. We will be using the following hexadecimal colors for the other classes: #FF0000 for developed, #1A11FF for water, and #D0741E for herbaceous. -![](F2/image41.png) +![Fig. F2.1.7 New layer option in Geometry Imports](F2/image41.png) -Fig. F2.1.7 New layer option in Geometry Imports -You should now have four FeatureCollection imports named forest, developed, water, and herbaceous (Fig. F2.1.8). +You should now have four FeatureCollection imports named forest, developed, water, and herbaceous (Fig. F2.1.8). -![](F2/image42.png) +![Fig. F2.1.8 Example of training points](F2/image42.png) -Fig. F2.1.8 Example of training points -::: {.callout-note} -Code Checkpoint F21a. The book’s repository contains a script that shows what your code should look like at this point. +:::{.callout-note} +Code Checkpoint F21a. The book’s repository contains a script that shows what your code should look like at this point. ::: If you wish to have the exact same results demonstrated in this chapter from now on, continue beginning with this Code Checkpoint. If you use the points collected yourself, the results may vary from this point forward. -The next step is to combine all the training feature collections into one. Copy and paste the code below to combine them into one FeatureCollection called trainingFeatures. Here, we use the flatten method to avoid having a collection of feature collections—we want individual features within our FeatureCollection. +The next step is to combine all the training feature collections into one. Copy and paste the code below to combine them into one FeatureCollection called trainingFeatures. Here, we use the flatten method to avoid having a collection of feature collections—we want individual features within our FeatureCollection. +```js // Combine training feature collections. -var trainingFeatures = ee.FeatureCollection([ -   forest, developed, water, herbaceous +var trainingFeatures = ee.FeatureCollection([ + forest, developed, water, herbaceous ]).flatten(); -Note: Alternatively, you could use an existing set of reference data. For example, the European Space Agency (ESA) WorldCover dataset is a global map of land use and land cover derived from ESA’s Sentinel-2 imagery at 10 m resolution. With existing datasets, we can randomly place points on pixels classified as the classes of interest (if you are curious, you can explore the Earth Engine documentation to learn about the ee.Image.stratifiedSample and the ee.FeatureCollection.randomPoints methods). The drawback is that these global datasets will not always contain the specific classes of interest for your region, or may not be entirely accurate at the local scale. Another option is to use samples that were collected in the field (e.g., GPS points). In Chap. F5.0, you will see how to upload your own data as Earth Engine assets. +``` +Note: Alternatively, you could use an existing set of reference data. For example, the European Space Agency (ESA) WorldCover dataset is a global map of land use and land cover derived from ESA’s Sentinel-2 imagery at 10 m resolution. With existing datasets, we can randomly place points on pixels classified as the classes of interest (if you are curious, you can explore the Earth Engine documentation to learn about the ee.Image.stratifiedSample and the ee.FeatureCollection.randomPoints methods). The drawback is that these global datasets will not always contain the specific classes of interest for your region, or may not be entirely accurate at the local scale. Another option is to use samples that were collected in the field (e.g., GPS points). In Chap. F5.0, you will see how to upload your own data as Earth Engine assets. -In the combined FeatureCollection, each Feature point should have a property called “class”. The class values are consecutive integers from 0 to 3 (you could verify that this is true by printing trainingFeatures and checking the properties of the features). +In the combined FeatureCollection, each Feature point should have a property called “class”. The class values are consecutive integers from 0 to 3 (you could verify that this is true by printing trainingFeatures and checking the properties of the features). -Now that we have our training points, copy and paste the code below to extract the band information for each class at each point location. First, we define the prediction bands to extract different spectral and thermal information from different bands for each class. Then, we use the sampleRegions method to sample the information from the Landsat image at each point location. This method requires information about the FeatureCollection (our reference points), the property to extract (“class”), and the pixel scale (in meters). +Now that we have our training points, copy and paste the code below to extract the band information for each class at each point location. First, we define the prediction bands to extract different spectral and thermal information from different bands for each class. Then, we use the sampleRegions method to sample the information from the Landsat image at each point location. This method requires information about the FeatureCollection (our reference points), the property to extract (“class”), and the pixel scale (in meters). +```js // Define prediction bands. -var predictionBands = [   'SR_B1', 'SR_B2', 'SR_B3', 'SR_B4', 'SR_B5', 'SR_B6', 'SR_B7',   'ST_B10' +var predictionBands = [ 'SR_B1', 'SR_B2', 'SR_B3', 'SR_B4', 'SR_B5', 'SR_B6', 'SR_B7', 'ST_B10' ]; // Sample training points. -var classifierTraining = landsat.select(predictionBands) -   .sampleRegions({ -       collection: trainingFeatures, -       properties: ['class'], -       scale: 30   }); +var classifierTraining = landsat.select(predictionBands) + .sampleRegions({ + collection: trainingFeatures, + properties: ['class'], + scale: 30 }); -You can check whether the classifierTraining object extracted the properties of interest by printing it and expanding the first feature. You should see the band and class information (Fig. F2.1.9). +``` +You can check whether the classifierTraining object extracted the properties of interest by printing it and expanding the first feature. You should see the band and class information (Fig. F2.1.9). -![](F2/image20.png) +![Fig. F2.1.9 Example of extracted band information for one point of class 0 (forest)](F2/image20.png) -Fig. F2.1.9 Example of extracted band information for one point of class 0 (forest) -Now we can choose a classifier. The choice of classifier is not always obvious, and there are many options from which to pick—you can quickly expand the ee.Classifier object under Docs to get an idea of how many options we have for image classification. Therefore, we will be testing different classifiers and comparing their results. We will start with a Classification and Regression Tree (CART) classifier, a well-known classification algorithm (Fig. F2.1.10) that has been around for decades. +Now we can choose a classifier. The choice of classifier is not always obvious, and there are many options from which to pick—you can quickly expand the ee.Classifier object under Docs to get an idea of how many options we have for image classification. Therefore, we will be testing different classifiers and comparing their results. We will start with a Classification and Regression Tree (CART) classifier, a well-known classification algorithm (Fig. F2.1.10) that has been around for decades. -![](F2/image25.png) +![Fig. F2.1.10 Example of a decision tree for satellite image classification. Values and classes are hypothetical.](F2/image25.png) -Fig. F2.1.10 Example of a decision tree for satellite image classification. Values and classes are hypothetical. -Copy and paste the code below to instantiate a CART classifier (ee.Classifier.smileCart) and train it. +Copy and paste the code below to instantiate a CART classifier (ee.Classifier.smileCart) and train it. +```js //////////////// CART Classifier /////////////////// // Train a CART Classifier. -var classifier = ee.Classifier.smileCart().train({ -   features: classifierTraining, -   classProperty: 'class', -   inputProperties: predictionBands +var classifier = ee.Classifier.smileCart().train({ + features: classifierTraining, + classProperty: 'class', + inputProperties: predictionBands }); -Essentially, the classifier contains the mathematical rules that link labels to spectral information. If you print the variable classifier and expand its properties, you can confirm the basic characteristics of the object (bands, properties, and classifier being used). If you print classifier.explain, you can find a property called “tree” that contains the decision rules. +``` +Essentially, the classifier contains the mathematical rules that link labels to spectral information. If you print the variable classifier and expand its properties, you can confirm the basic characteristics of the object (bands, properties, and classifier being used). If you print classifier.explain, you can find a property called “tree” that contains the decision rules. After training the classifier, copy and paste the code below to classify the Landsat image and add it to the Map. +```js // Classify the Landsat image. -var classified = landsat.select(predictionBands).classify(classifier); +var classified = landsat.select(predictionBands).classify(classifier); // Define classification image visualization parameters. -var classificationVis = { -   min: 0, -   max: 3, -   palette: ['589400', 'ff0000', '1a11ff', 'd0741e'] +var classificationVis = { + min: 0, + max: 3, + palette: ['589400', 'ff0000', '1a11ff', 'd0741e'] }; // Add the classified image to the map. Map.addLayer(classified, classificationVis, 'CART classified'); -Note that, in the visualization parameters, we define a palette parameter which in this case represents colors for each pixel value (0–3, our class codes). We use the same hexadecimal colors used when creating our training points for each class. This way, we can associate a color with a class when visualizing the classified image in the Map. +``` +Note that, in the visualization parameters, we define a palette parameter which in this case represents colors for each pixel value (0–3, our class codes). We use the same hexadecimal colors used when creating our training points for each class. This way, we can associate a color with a class when visualizing the classified image in the Map. -Inspect the result: Activate the Landsat composite layer and the satellite basemap to overlay with the classified images (Fig. F2.1.11). Change the layers’ transparency to inspect some areas. What do you notice? The result might not look very satisfactory in some areas (e.g., confusion between developed and herbaceous classes). Why do you think this is happening? There are a few options to handle misclassification errors: +Inspect the result: Activate the Landsat composite layer and the satellite basemap to overlay with the classified images (Fig. F2.1.11). Change the layers’ transparency to inspect some areas. What do you notice? The result might not look very satisfactory in some areas (e.g., confusion between developed and herbaceous classes). Why do you think this is happening? There are a few options to handle misclassification errors: -* Collect more training data We can try incorporating more points to have a more representative sample of the classes. -* Tune the model Classifiers typically have “hyperparameters,” which are set to default values. In the case of classification trees, there are ways to tune the number of leaves in the tree, for example. Tuning models is addressed in Chap. F2.2. -* Try other classifiers If a classifier’s results are unsatisfying, we can try some of the other classifiers in Earth Engine to see if the result is better or different. -* Expand the collection location It is good practice to collect points across the entire image and not just focus on one location. Also, look for pixels of the same class that show variability (e.g., for the developed class, building rooftops look different than house rooftops; for the herbaceous class, crop fields show distinctive seasonality/phenology).  -* Add more predictors We can try adding spectral indices to the input variables; this way, we are feeding the classifier new, unique information about each class. For example, there is a good chance that a vegetation index specialized for detecting vegetation health (e.g., NDVI) would improve the developed versus herbaceous classification. +* Collect more training data We can try incorporating more points to have a more representative sample of the classes. +* Tune the model Classifiers typically have “hyperparameters,” which are set to default values. In the case of classification trees, there are ways to tune the number of leaves in the tree, for example. Tuning models is addressed in Chap. F2.2. +* Try other classifiers If a classifier’s results are unsatisfying, we can try some of the other classifiers in Earth Engine to see if the result is better or different. +* Expand the collection location It is good practice to collect points across the entire image and not just focus on one location. Also, look for pixels of the same class that show variability (e.g., for the developed class, building rooftops look different than house rooftops; for the herbaceous class, crop fields show distinctive seasonality/phenology). +* Add more predictors We can try adding spectral indices to the input variables; this way, we are feeding the classifier new, unique information about each class. For example, there is a good chance that a vegetation index specialized for detecting vegetation health (e.g., NDVI) would improve the developed versus herbaceous classification. -![](F2/image21.png) +![Fig. F2.1.11 CART classification](F2/image21.png) -Fig. F2.1.11 CART classification -For now, we will try another supervised learning classifier that is widely used: Random Forests (RF). The RF algorithm (Breiman 2001, Pal 2005) builds on the concept of decision trees, but adds strategies to make them more powerful. It is called a “forest” because it operates by constructing a multitude of decision trees. As mentioned previously, a decision tree creates the rules which are used to make decisions. A Random Forest will randomly choose features and make observations, build a forest of decision trees, and then use the full set of trees to estimate the class. It is a great choice when you do not have a lot of insight about the training data. +For now, we will try another supervised learning classifier that is widely used: Random Forests (RF). The RF algorithm (Breiman 2001, Pal 2005) builds on the concept of decision trees, but adds strategies to make them more powerful. It is called a “forest” because it operates by constructing a multitude of decision trees. As mentioned previously, a decision tree creates the rules which are used to make decisions. A Random Forest will randomly choose features and make observations, build a forest of decision trees, and then use the full set of trees to estimate the class. It is a great choice when you do not have a lot of insight about the training data. -![](F2/image27.png) +![Fig. F2.1.12 General concept of Random Forests](F2/image27.png) -Fig. F2.1.12 General concept of Random Forests Copy and paste the code below to train the RF classifier (ee.Classifier.smileRandomForest) and apply the classifier to the image. The RF algorithm requires, as its argument, the number of trees to build. We will use 50 trees. +```js /////////////// Random Forest Classifier ///////////////////// // Train RF classifier. -var RFclassifier = ee.Classifier.smileRandomForest(50).train({ -   features: classifierTraining, -   classProperty: 'class', -   inputProperties: predictionBands +var RFclassifier = ee.Classifier.smileRandomForest(50).train({ + features: classifierTraining, + classProperty: 'class', + inputProperties: predictionBands }); // Classify Landsat image. -var RFclassified = landsat.select(predictionBands).classify( -   RFclassifier); +var RFclassified = landsat.select(predictionBands).classify( + RFclassifier); // Add classified image to the map. Map.addLayer(RFclassified, classificationVis, 'RF classified'); -Note that in the ee.Classifier.smileRandomForest documentation (Docs tab), there is a seed (random number) parameter. Setting a seed allows you to exactly replicate your model each time you run it. Any number is acceptable as a seed. +``` +Note that in the ee.Classifier.smileRandomForest documentation (Docs tab), there is a seed (random number) parameter. Setting a seed allows you to exactly replicate your model each time you run it. Any number is acceptable as a seed. Inspect the result (Fig. F2.1.13). How does this classified image differ from the CART one? Is the classifications better or worse? Zoom in and out and change the transparency of layers as needed. In Chap. F2.2, you will see more systematic ways to assess what is better or worse, based on accuracy metrics. -![](F2/image34.png) +![Fig. F2.1.13 Random Forest classified image](F2/image34.png) -Fig. F2.1.13 Random Forest classified image -::: {.callout-note} -Code Checkpoint F21b. The book’s repository contains a script that shows what your code should look like at this point. +:::{.callout-note} +Code Checkpoint F21b. The book’s repository contains a script that shows what your code should look like at this point. ::: ## Unsupervised Classification -In an unsupervised classification, we have the opposite process of supervised classification. Spectral classes are grouped first and then categorized into clusters. Therefore, in Earth Engine, these classifiers are ee.Clusterer objects. They are “self-taught” algorithms that do not use a set of labeled training data (i.e., they are “unsupervised”). You can think of it as performing a task that you have not experienced before, starting by gathering as much information as possible. For example, imagine learning a new language without knowing the basic grammar, learning only by watching a TV series in that language, listening to examples, and finding patterns. +In an unsupervised classification, we have the opposite process of supervised classification. Spectral classes are grouped first and then categorized into clusters. Therefore, in Earth Engine, these classifiers are ee.Clusterer objects. They are “self-taught” algorithms that do not use a set of labeled training data (i.e., they are “unsupervised”). You can think of it as performing a task that you have not experienced before, starting by gathering as much information as possible. For example, imagine learning a new language without knowing the basic grammar, learning only by watching a TV series in that language, listening to examples, and finding patterns. -Similar to the supervised classification, unsupervised classification in Earth Engine has this workflow: +Similar to the supervised classification, unsupervised classification in Earth Engine has this workflow: 1. Assemble features with numeric properties in which to find clusters (training data). 2. Select and instantiate a clusterer. @@ -684,47 +691,51 @@ Similar to the supervised classification, unsupervised classification in Earth E 4. Apply the clusterer to the scene (classification). 5. Label the clusters. -In order to generate training data, we will use the sample method, which randomly takes samples from a region (unlike sampleRegions, which takes samples from predefined locations). We will use the image’s footprint as the region by calling the geometry method. Additionally, we will define the number of pixels (numPixels) to sample—in this case, 1000 pixels—and define a tileScale of 8 to avoid computation errors due to the size of the region. Copy and paste the code below to sample 1000 pixels from the Landsat image. You should add to the same script as before to compare supervised versus unsupervised classification results at the end. +In order to generate training data, we will use the sample method, which randomly takes samples from a region (unlike sampleRegions, which takes samples from predefined locations). We will use the image’s footprint as the region by calling the geometry method. Additionally, we will define the number of pixels (numPixels) to sample—in this case, 1000 pixels—and define a tileScale of 8 to avoid computation errors due to the size of the region. Copy and paste the code below to sample 1000 pixels from the Landsat image. You should add to the same script as before to compare supervised versus unsupervised classification results at the end. +```js //////////////// Unsupervised classification //////////////// // Make the training dataset. -var training = landsat.sample({ -   region: landsat.geometry(), -   scale: 30, -   numPixels: 1000, -   tileScale: 8 +var training = landsat.sample({ + region: landsat.geometry(), + scale: 30, + numPixels: 1000, + tileScale: 8 }); +``` Now we can instantiate a clusterer and train it. As with the supervised algorithms, there are many unsupervised algorithms to choose from. We will use the k-means clustering algorithm, which is a commonly used approach in remote sensing. This algorithm identifies groups of pixels near each other in the spectral space (image x bands) by using an iterative regrouping strategy. We define a number of clusters, k, and then the method randomly distributes that number of seed points into the spectral space. A large sample of pixels is then grouped into its closest seed, and the mean spectral value of this group is calculated. That mean value is akin to a center of mass of the points, and is known as the centroid. Each iteration recalculates the class means and reclassifies pixels with respect to the new means. This process is repeated until the centroids remain relatively stable and only a few pixels change from class to class on subsequent iterations. -![](F2/image35.png) +![Fig. F2.1.14 K-means visual concept](F2/image35.png) -Fig. F2.1.14 K-means visual concept -Copy and paste the code below to request four clusters, the same number as for the supervised classification, in order to directly compare them. +Copy and paste the code below to request four clusters, the same number as for the supervised classification, in order to directly compare them. +```js // Instantiate the clusterer and train it. -var clusterer = ee.Clusterer.wekaKMeans(4).train(training); +var clusterer = ee.Clusterer.wekaKMeans(4).train(training); -Now copy and paste the code below to apply the clusterer to the image and add the resulting classification to the Map (Fig. F2.1.15). Note that we are using a method called randomVisualizer to assign colors for the visualization. We are not associating the unsupervised classes with the color palette we defined earlier in the supervised classification. Instead, we are assigning random colors to the classes, since we do not yet know which of the unsupervised classes best corresponds to each of the named classes (e.g., forest , herbaceous). Note that the colors in Fig. F1.2.15 might not be the same as you see on your Map, since they are assigned randomly. +``` +Now copy and paste the code below to apply the clusterer to the image and add the resulting classification to the Map (Fig. F2.1.15). Note that we are using a method called randomVisualizer to assign colors for the visualization. We are not associating the unsupervised classes with the color palette we defined earlier in the supervised classification. Instead, we are assigning random colors to the classes, since we do not yet know which of the unsupervised classes best corresponds to each of the named classes (e.g., forest , herbaceous). Note that the colors in Fig. F1.2.15 might not be the same as you see on your Map, since they are assigned randomly. +```js // Cluster the input using the trained clusterer. -var Kclassified = landsat.cluster(clusterer); +var Kclassified = landsat.cluster(clusterer); // Display the clusters with random colors. -Map.addLayer(Kclassified.randomVisualizer(), {},   'K-means classified - random colors'); +Map.addLayer(Kclassified.randomVisualizer(), {}, 'K-means classified - random colors'); -![](F2/image31.png) +``` +![Fig. F2.1.15 K-means classification](F2/image31.png) -Fig. F2.1.15 K-means classification -Inspect the results. How does this classification compare to the previous ones? If preferred, use the Inspector to check which classes were assigned to each pixel value (“cluster” band) and change the last line of your code to apply the same palette used for the supervised classification results (see Code Checkpoint below for an example). +Inspect the results. How does this classification compare to the previous ones? If preferred, use the Inspector to check which classes were assigned to each pixel value (“cluster” band) and change the last line of your code to apply the same palette used for the supervised classification results (see Code Checkpoint below for an example). Another key point of classification is the accuracy assessment of the results. This will be covered in Chap. F2.2. -::: {.callout-note} -Code Checkpoint F21c. The book’s repository contains a script that shows what your code should look like at this point. +:::{.callout-note} +Code Checkpoint F21c. The book’s repository contains a script that shows what your code should look like at this point. ::: ## Synthesis {.unnumbered} @@ -732,15 +743,15 @@ Test if you can improve the classifications by completing the following assignme Assignment 1. For the supervised classification, try collecting more points for each class. The more points you have, the more spectrally represented the classes are. It is good practice to collect points across the entire composite and not just focus on one location. Also look for pixels of the same class that show variability. For example, for the water class, collect pixels in parts of rivers that vary in color. For the developed class, collect pixels from different rooftops. -Assignment 2. Add more predictors. Usually, the more spectral information you feed the classifier, the easier it is to separate classes. Try calculating and incorporating a band of  NDVI or the Normalized Difference Water Index (Chap. F2.0) as a predictor band. Does this help the classification? Check for developed areas that were being classified as herbaceous or vice versa. +Assignment 2. Add more predictors. Usually, the more spectral information you feed the classifier, the easier it is to separate classes. Try calculating and incorporating a band of NDVI or the Normalized Difference Water Index (Chap. F2.0) as a predictor band. Does this help the classification? Check for developed areas that were being classified as herbaceous or vice versa. -Assignment 3. Use more trees in the Random Forest classifier. Do you see any improvements compared to 50 trees? Note that the more trees you have, the longer it will take to compute the results, and that more trees might not always mean better results. +Assignment 3. Use more trees in the Random Forest classifier. Do you see any improvements compared to 50 trees? Note that the more trees you have, the longer it will take to compute the results, and that more trees might not always mean better results. Assignment 4. Increase the number of samples that are extracted from the composite in the unsupervised classification. Does that improve the result? -Assignment 5. Increase the number k of clusters for the k-means algorithm. What would happen if you tried 10 classes? Does the classified map result in meaningful classes? +Assignment 5. Increase the number k of clusters for the k-means algorithm. What would happen if you tried 10 classes? Does the classified map result in meaningful classes? -Assignment 6. Test other clustering algorithms. We only used k-means; try other options under the ee.Clusterer object. +Assignment 6. Test other clustering algorithms. We only used k-means; try other options under the ee.Clusterer object. ## Conclusion {.unnumbered} @@ -774,7 +785,7 @@ Witten IH, Frank E, Hall MA, et al (2005) Practical machine learning tools and t -::: {.callout-tip} +:::{.callout-tip} # Chapter Information ## Author {.unlisted .unnumbered} @@ -801,10 +812,10 @@ This chapter will enable you to assess the accuracy of an image classification. ## Assumes you know how to:{.unlisted .unnumbered} -* ​​Create a graph using ui.Chart (Chap. F1.3). +* ​​Create a graph using ui.Chart (Chap. F1.3). * Perform a supervised Random Forest image classification (Chap. F2.1). -::: +::: ## Introduction {.unlisted .unnumbered} @@ -813,17 +824,17 @@ Any map or remotely sensed product is a generalization or model that will have i The history of accuracy assessment reveals increasing detail and rigor in the analysis, moving from a basic visual appraisal of the derived map (Congalton 1994, Foody 2002) to the definition of best practices for sampling and response designs and the calculation of accuracy metrics (Foody 2002, Stehman 2013, Olofsson et al. 2014, Stehman and Foody 2019). The confusion matrix (also called the “error matrix”) (Stehman 1997) summarizes key accuracy metrics used to assess products derived from remotely sensed data. -In Chap. F2.1, we asked whether the classification results were satisfactory. In remote sensing, the quantification of the answer to that question is called accuracy assessment. In the classification context, accuracy measurements are often derived from a confusion matrix. +In Chap. F2.1, we asked whether the classification results were satisfactory. In remote sensing, the quantification of the answer to that question is called accuracy assessment. In the classification context, accuracy measurements are often derived from a confusion matrix. -In a thorough accuracy assessment, we think carefully about the sampling design, the response design, and the analysis (Olofsson et al. 2014). Fundamental protocols are taken into account to produce scientifically rigorous and transparent estimates of accuracy and area, which requires robust planning and time. In a standard setting, we would calculate the number of samples needed for measuring accuracy (sampling design). Here, we will focus mainly on the last step, analysis, by examining the confusion matrix and learning how to calculate the accuracy metrics. This will be done by partitioning the existing data into training and testing sets. +In a thorough accuracy assessment, we think carefully about the sampling design, the response design, and the analysis (Olofsson et al. 2014). Fundamental protocols are taken into account to produce scientifically rigorous and transparent estimates of accuracy and area, which requires robust planning and time. In a standard setting, we would calculate the number of samples needed for measuring accuracy (sampling design). Here, we will focus mainly on the last step, analysis, by examining the confusion matrix and learning how to calculate the accuracy metrics. This will be done by partitioning the existing data into training and testing sets. ## Quantifying Classification Accuracy Through a Confusion Matrix -If you have not already done so, be sure to add the book’s code repository to the Code Editor by entering [](https://www.google.com/url?q=https://code.earthengine.google.com/?accept_repo%3Dprojects/gee-edu/book&sa=D&source=editors&ust=1671458829937499&usg=AOvVaw3qqOwSX_A-Pllh6X3X31q4)[https://code.earthengine.google.com/?accept_repo=projects/gee-edu/book](https://www.google.com/url?q=https://code.earthengine.google.com/?accept_repo%3Dprojects/gee-edu/book&sa=D&source=editors&ust=1671458829937976&usg=AOvVaw0WioXIhzue8-WoaX4UtabH) into your browser. The book’s scripts will then be available in the script manager panel. If you have trouble finding the repo, you can visit [this link](https://www.google.com/url?q=https://docs.google.com/presentation/d/1Kt6wGNoesYm__Cu3k3bnlbbyPN6m9SF4hQHK-pIDHfc/edit%23slide%3Did.g18a7b4b055d_0_624&sa=D&source=editors&ust=1671458829938470&usg=AOvVaw2CH8V3-_qV99EcgMxUAaSO) for help. +If you have not already done so, be sure to add the book’s code repository to the Code Editor by entering [](https://www.google.com/url?q=https://code.earthengine.google.com/?accept_repo%3Dprojects/gee-edu/book&sa=D&source=editors&ust=1671458829937499&usg=AOvVaw3qqOwSX_A-Pllh6X3X31q4)[https://code.earthengine.google.com/?accept_repo=projects/gee-edu/book](https://www.google.com/url?q=https://code.earthengine.google.com/?accept_repo%3Dprojects/gee-edu/book&sa=D&source=editors&ust=1671458829937976&usg=AOvVaw0WioXIhzue8-WoaX4UtabH) into your browser. The book’s scripts will then be available in the script manager panel. If you have trouble finding the repo, you can visit [this link](https://www.google.com/url?q=https://docs.google.com/presentation/d/1Kt6wGNoesYm__Cu3k3bnlbbyPN6m9SF4hQHK-pIDHfc/edit%23slide%3Did.g18a7b4b055d_0_624&sa=D&source=editors&ust=1671458829938470&usg=AOvVaw2CH8V3-_qV99EcgMxUAaSO) for help. -To illustrate some of the basic ideas about classification accuracy, we will revisit the data and location of part of Chap. F2.1, where we tested different classifiers and classified a Landsat image of the area around Milan, Italy. We will name this dataset 'data'. This variable is a FeatureCollection with features containing the “class” values (Table F2.2.1) and spectral information of four land cover / land use classes: forest, developed, water, and herbaceous (see Fig. F2.1.8 and Fig. F2.1.9 for a refresher). We will also define a variable, predictionBands, which is a list of bands that will be used for prediction (classification)—the spectral information in the data variable. +To illustrate some of the basic ideas about classification accuracy, we will revisit the data and location of part of Chap. F2.1, where we tested different classifiers and classified a Landsat image of the area around Milan, Italy. We will name this dataset 'data'. This variable is a FeatureCollection with features containing the “class” values (Table F2.2.1) and spectral information of four land cover / land use classes: forest, developed, water, and herbaceous (see Fig. F2.1.8 and Fig. F2.1.9 for a refresher). We will also define a variable, predictionBands, which is a list of bands that will be used for prediction (classification)—the spectral information in the data variable. -Table F2.2.1 Land cover classes +Table F2.2.1 Land cover classes Class @@ -845,36 +856,40 @@ Herbaceous 3 -The first step is to partition the set of known values into training and testing sets in order to have something for the classifier to predict over that it has not been shown before (the testing set), mimicking unseen data that the model might see in the future. We add a column of random numbers to our FeatureCollection using the randomColumn method. Then, we filter the features into about 80% for training and 20% for testing using ee.Filter. Copy and paste the code below to partition the data and filter features based on the random number. +The first step is to partition the set of known values into training and testing sets in order to have something for the classifier to predict over that it has not been shown before (the testing set), mimicking unseen data that the model might see in the future. We add a column of random numbers to our FeatureCollection using the randomColumn method. Then, we filter the features into about 80% for training and 20% for testing using ee.Filter. Copy and paste the code below to partition the data and filter features based on the random number. +```js // Import the reference dataset. -var data = ee.FeatureCollection(   'projects/gee-book/assets/F2-2/milan_data'); +var data = ee.FeatureCollection( 'projects/gee-book/assets/F2-2/milan_data'); // Define the prediction bands. -var predictionBands = [   'SR_B1', 'SR_B2', 'SR_B3', 'SR_B4', 'SR_B5', 'SR_B6', 'SR_B7',   'ST_B10',   'ndvi', 'ndwi' +var predictionBands = [ 'SR_B1', 'SR_B2', 'SR_B3', 'SR_B4', 'SR_B5', 'SR_B6', 'SR_B7', 'ST_B10', 'ndvi', 'ndwi' ]; // Split the dataset into training and testing sets. -var trainingTesting = data.randomColumn(); -var trainingSet = trainingTesting -   .filter(ee.Filter.lessThan('random', 0.8)); -var testingSet = trainingTesting -   .filter(ee.Filter.greaterThanOrEquals('random', 0.8)); +var trainingTesting = data.randomColumn(); +var trainingSet = trainingTesting + .filter(ee.Filter.lessThan('random', 0.8)); +var testingSet = trainingTesting + .filter(ee.Filter.greaterThanOrEquals('random', 0.8)); -Note that randomColumn creates pseudorandom numbers in a deterministic way. This makes it possible to generate a reproducible pseudorandom sequence by defining the seed parameter (Earth Engine uses a seed of 0 by default). In other words, given a starting value (i.e., the seed), randomColumn will always provide the same sequence of pseudorandom numbers. +``` +Note that randomColumn creates pseudorandom numbers in a deterministic way. This makes it possible to generate a reproducible pseudorandom sequence by defining the seed parameter (Earth Engine uses a seed of 0 by default). In other words, given a starting value (i.e., the seed), randomColumn will always provide the same sequence of pseudorandom numbers. Copy and paste the code below to train a Random Forest classifier with 50 decision trees using the trainingSet. +```js // Train the Random Forest Classifier with the trainingSet. -var RFclassifier = ee.Classifier.smileRandomForest(50).train({ -   features: trainingSet, -   classProperty: 'class', -   inputProperties: predictionBands +var RFclassifier = ee.Classifier.smileRandomForest(50).train({ + features: trainingSet, + classProperty: 'class', + inputProperties: predictionBands }); +``` Now, let’s discuss what a confusion matrix is. A confusion matrix describes the quality of a classification by comparing the predicted values to the actual values. A simple example is a confusion matrix for a binary classification into the classes “positive” and “negative,” as shown in Table F2.2.1. -Table F2.2.1 Confusion matrix for a binary classification where the classes are “positive” and “negative” +Table F2.2.1 Confusion matrix for a binary classification where the classes are “positive” and “negative” Actual values @@ -886,15 +901,15 @@ Predicted values Positive -TP (true positive) +TP (true positive) -FP (false positive) +FP (false positive) Negative -FN (false negative) +FN (false negative) -TN (true negative) +TN (true negative) In Table F2.2.1, the columns represent the actual values (the truth), while the rows represent the predictions (the classification). “True positive” (TP) and “true negative” (TN) mean that the classification of a pixel matches the truth (e.g., a water pixel correctly classified as water). “False positive” (FP) and “false negative” (FN) mean that the classification of a pixel does not match the truth (e.g., a non-water pixel incorrectly classified as water). @@ -905,7 +920,7 @@ In Table F2.2.1, the columns represent the actual values (the truth), while the We can extract some statistical information from a confusion matrix.. Let’s look at an example to make this clearer. Table F2.2.2 is a confusion matrix for a sample of 1,000 pixels for a classifier that identifies whether a pixel is forest (positive) or non-forest (negative), a binary classification. -Table F2.2.2 Confusion matrix for a binary classification where the classes are “positive” (forest) and “negative” (non-forest) +Table F2.2.2 Confusion matrix for a binary classification where the classes are “positive” (forest) and “negative” (non-forest) Actual values @@ -951,23 +966,22 @@ The user’s accuracy (also called the “consumer’s accuracy”) is the accur ![](F2/image13.png) -In this case, the user’s accuracy for the forest class is 94.5%, calculated using ![](F2/image14.png)). The user’s accuracy for the non-forest class is 97.9%, calculated from ![](F2/image15.png)). +In this case, the user’s accuracy for the forest class is 94.5%, calculated using ![](F2/image14.png)). The user’s accuracy for the non-forest class is 97.9%, calculated from ![](F2/image15.png)). Fig. F2.2.1 helps visualize the rows and columns used to calculate each accuracy. -![](F2/image43.png) +![Fig. F2.2.1 Confusion matrix for a binary classification where the classes are “positive” (forest) and “negative” (non-forest), with accuracy metrics](F2/image43.png) -Fig. F2.2.1 Confusion matrix for a binary classification where the classes are “positive” (forest) and “negative” (non-forest), with accuracy metrics It is very common to talk about two types of error when addressing remote-sensing classification accuracy: omission errors and commission errors. Omission errors refer to the reference pixels that were left out of (omitted from) the correct class in the classified map. In a two-class system, an error of omission in one class will be counted as an error of commission in another class. Omission errors are complementary to the producer’s accuracy. -![](F2/image16.png)  +![](F2/image16.png) Commission errors refer to the class pixels that were erroneously classified in the map and are complementary to the user’s accuracy. ![](F2/image17.png) -Finally, another commonly used accuracy metric is the kappa coefficient, which  evaluates how well the classification performed as compared to random. The value of the kappa coefficient can range from −1 to 1: a negative value indicates that the classification is worse than a random assignment of categories would have been; a value of 0 indicates that the classification is no better or worse than random; and a positive value indicates that the classification is better than random. +Finally, another commonly used accuracy metric is the kappa coefficient, which evaluates how well the classification performed as compared to random. The value of the kappa coefficient can range from −1 to 1: a negative value indicates that the classification is worse than a random assignment of categories would have been; a value of 0 indicates that the classification is no better or worse than random; and a positive value indicates that the classification is better than random. ![](F2/image18.png) @@ -977,17 +991,20 @@ The chance agreement is calculated as the sum of the product of row and column t Now, let’s go back to the script. In Earth Engine, there are API calls for these operations. Note that our confusion matrix will be a 4 x 4 table, since we have four different classes. -Copy and paste the code below to classify the testingSet and get a confusion matrix using the method errorMatrix. Note that the classifier automatically adds a property called “classification,” which is compared to the “class” property of the reference dataset. +Copy and paste the code below to classify the testingSet and get a confusion matrix using the method errorMatrix. Note that the classifier automatically adds a property called “classification,” which is compared to the “class” property of the reference dataset. +```js // Now, to test the classification (verify model's accuracy), // we classify the testingSet and get a confusion matrix. -var confusionMatrix = testingSet.classify(RFclassifier) -   .errorMatrix({ -       actual: 'class', -       predicted: 'classification'   }); +var confusionMatrix = testingSet.classify(RFclassifier) + .errorMatrix({ + actual: 'class', + predicted: 'classification' }); -Copy and paste the code below to print the confusion matrix and accuracy metrics. Expand the confusion matrix object to inspect it. The entries represent the number of pixels. Items on the diagonal represent correct classification. Items off the diagonal are misclassifications, where the class in row i is classified as column j (values from 0 to 3 correspond to our class codes: forest, developed, water, and herbaceous, respectively). Also expand the producer’s accuracy, user’s accuracy (consumer’s accuracy), and kappa coefficient objects to inspect them. +``` +Copy and paste the code below to print the confusion matrix and accuracy metrics. Expand the confusion matrix object to inspect it. The entries represent the number of pixels. Items on the diagonal represent correct classification. Items off the diagonal are misclassifications, where the class in row i is classified as column j (values from 0 to 3 correspond to our class codes: forest, developed, water, and herbaceous, respectively). Also expand the producer’s accuracy, user’s accuracy (consumer’s accuracy), and kappa coefficient objects to inspect them. +```js // Print the results. print('Confusion matrix:', confusionMatrix); print('Overall Accuracy:', confusionMatrix.accuracy()); @@ -995,66 +1012,68 @@ print('Producers Accuracy:', confusionMatrix.producersAccuracy()); print('Consumers Accuracy:', confusionMatrix.consumersAccuracy()); print('Kappa:', confusionMatrix.kappa()); -How is the classification accuracy? Which classes have higher accuracy compared to the others? Can you think of any reasons why? (Hint: Check where the errors in these classes are in the confusion matrix—i.e., being committed and omitted.) +``` +How is the classification accuracy? Which classes have higher accuracy compared to the others? Can you think of any reasons why? (Hint: Check where the errors in these classes are in the confusion matrix—i.e., being committed and omitted.) -::: {.callout-note} -Code Checkpoint F22a. The book’s repository contains a script that shows what your code should look like at this point. +:::{.callout-note} +Code Checkpoint F22a. The book’s repository contains a script that shows what your code should look like at this point. ::: ## Hyperparameter tuning We can also assess how the number of trees in the Random Forest classifier affects the classification accuracy. Copy and paste the code below to create a function that charts the overall accuracy versus the number of trees used. The code tests from 5 to 100 trees at increments of 5, producing Fig. F2.2.2. (Do not worry too much about fully understanding each item at this stage of your learning. If you want to find out how these operations work, you can see more in Chaps. F4.0 and F4.1.) +```js // Hyperparameter tuning. -var numTrees = ee.List.sequence(5, 100, 5); +var numTrees = ee.List.sequence(5, 100, 5); -var accuracies = numTrees.map(function(t) {   var classifier = ee.Classifier.smileRandomForest(t) -       .train({ -           features: trainingSet, -           classProperty: 'class', -           inputProperties: predictionBands -       });   return testingSet -       .classify(classifier) -       .errorMatrix('class', 'classification') -       .accuracy(); +var accuracies = numTrees.map(function(t) { var classifier = ee.Classifier.smileRandomForest(t) + .train({ + features: trainingSet, + classProperty: 'class', + inputProperties: predictionBands + }); return testingSet + .classify(classifier) + .errorMatrix('class', 'classification') + .accuracy(); }); print(ui.Chart.array.values({ -   array: ee.Array(accuracies), -   axis: 0, -   xLabels: numTrees + array: ee.Array(accuracies), + axis: 0, + xLabels: numTrees }).setOptions({ -   hAxis: { -       title: 'Number of trees'   }, -   vAxis: { -       title: 'Accuracy'   }, -   title: 'Accuracy per number of trees' + hAxis: { + title: 'Number of trees' }, + vAxis: { + title: 'Accuracy' }, + title: 'Accuracy per number of trees' })); -![](F2/image45.png) +``` +![Fig. F2.2.2 Chart showing accuracy per number of Random Forest trees](F2/image45.png) -Fig. F2.2.2 Chart showing accuracy per number of Random Forest trees -::: {.callout-note} -Code Checkpoint F22b. The book’s repository contains a script that shows what your code should look like at this point. +:::{.callout-note} +Code Checkpoint F22b. The book’s repository contains a script that shows what your code should look like at this point. ::: -Section 3. Spatial autocorrelation +Section 3. Spatial autocorrelation We might also want to ensure that the samples from the training set are uncorrelated with the samples from the testing set. This might result from the spatial autocorrelation of the phenomenon being predicted. One way to exclude samples that might be correlated in this manner is to remove samples that are within some distance to any other sample. In Earth Engine, this can be accomplished with a spatial join. The following Code Checkpoint replicates Sect. 1 but with a spatial join that excludes training points that are less than 1000 meters distant from testing points. -::: {.callout-note} -Code Checkpoint F22c. The book’s repository contains a script that shows what your code should look like at this point. +:::{.callout-note} +Code Checkpoint F22c. The book’s repository contains a script that shows what your code should look like at this point. ::: ## Synthesis {.unnumbered} -Assignment 1. Based on Sect. 1, test other classifiers (e.g., a Classification and Regression Tree or Support Vector Machine classifier) and compare the accuracy results with the Random Forest results. Which model performs better? +Assignment 1. Based on Sect. 1, test other classifiers (e.g., a Classification and Regression Tree or Support Vector Machine classifier) and compare the accuracy results with the Random Forest results. Which model performs better? -Assignment 2. Try setting a different seed in the randomColumn method and see how that affects the accuracy results. You can also change the split between the training and testing sets (e.g., 70/30 or 60/40). +Assignment 2. Try setting a different seed in the randomColumn method and see how that affects the accuracy results. You can also change the split between the training and testing sets (e.g., 70/30 or 60/40). ## Conclusion {.unnumbered} You should now understand how to calculate how well your classifier is performing on the data used to build the model. This is a useful way to understand how a classifier is performing, because it can help indicate which classes are performing better than others. A poorly modeled class can sometimes be improved by, for example, collecting more training points for that class. -Nevertheless, a model may work well on training data but work poorly in locations randomly chosen in the study area. To understand a model’s behavior on testing data, analysts employ protocols required to produce scientifically rigorous and transparent estimates of the accuracy and area of each class in the study region. We will not explore those practices in this chapter, but if you are interested, there are tutorials and papers available online that can guide you through the process. Links to some of those tutorials can be found in the “For Further Reading” section of this book. +Nevertheless, a model may work well on training data but work poorly in locations randomly chosen in the study area. To understand a model’s behavior on testing data, analysts employ protocols required to produce scientifically rigorous and transparent estimates of the accuracy and area of each class in the study region. We will not explore those practices in this chapter, but if you are interested, there are tutorials and papers available online that can guide you through the process. Links to some of those tutorials can be found in the “For Further Reading” section of this book. References diff --git a/F4.qmd b/F4.qmd index f7eca4a..cf10e7c 100644 --- a/F4.qmd +++ b/F4.qmd @@ -1,12 +1,12 @@ # Interpreting Image Series -One of the paradigm-changing features of Earth Engine is the ability to access decades of imagery without the previous limitation of needing to download all the data to a local disk for processing. Because remote-sensing data files can be enormous, this used to limit many projects to viewing two or three images from different periods. With Earth Engine, users can access tens or hundreds of thousands of images to understand the status of places across decades. +One of the paradigm-changing features of Earth Engine is the ability to access decades of imagery without the previous limitation of needing to download all the data to a local disk for processing. Because remote-sensing data files can be enormous, this used to limit many projects to viewing two or three images from different periods. With Earth Engine, users can access tens or hundreds of thousands of images to understand the status of places across decades. # Filter, Map, Reduce -::: {.callout-tip} +:::{.callout-tip} # Chapter Information ## Author {.unlisted .unnumbered} @@ -22,7 +22,7 @@ The purpose of this chapter is to teach you important programming concepts as th ## Learning Outcomes {.unlisted .unnumbered} -* Visualizing the concepts of filtering, mapping, and reducing with a hypothetical, non-programming example. +* Visualizing the concepts of filtering, mapping, and reducing with a hypothetical, non-programming example. * Gaining context and experience with filtering an ImageCollection. * Learning how to efficiently map a user-written function over the images of a filtered ImageCollection. * Learning how to summarize a set of assembled values using Earth Engine reducers. @@ -33,7 +33,7 @@ The purpose of this chapter is to teach you important programming concepts as th * Import images and image collections, filter, and visualize (Part F1). * Perform basic image analysis: select bands, compute indices, create masks (Part F2). -::: +::: ## Introduction {.unlisted .unnumbered} @@ -41,7 +41,7 @@ Prior chapters focused on exploring individual images—for example, viewing the In this chapter and most of the chapters that follow, we will move from the domain of single images to the more complex and distinctive world of working with image collections, one of the fundamental data types within Earth Engine. The ability to conceptualize and manipulate entire image collections distinguishes Earth Engine and gives it considerable power for interpreting change and stability across space and time. -When looking for change or seeking to understand differences in an area through time, we often proceed through three ordered stages, which we will color code in this first explanatory part of the lab: +When looking for change or seeking to understand differences in an area through time, we often proceed through three ordered stages, which we will color code in this first explanatory part of the lab: 1. Filter: selecting subsets of images based on criteria of interest. 2. Map: manipulating each image in a set in some way to suit our goals. and @@ -49,19 +49,19 @@ When looking for change or seeking to understand differences in an area through For users of other programming languages—R, MATLAB, C, Karel, and many others—this approach might seem awkward at first. We explain it below with a non-programming example: going to the store to buy milk. -Suppose you need to go shopping for milk, and you have two criteria for determining where you will buy your milk: location and price. The store needs to be close to your home, and as a first step in deciding whether to buy milk today, you want to identify the lowest price among those stores. You don’t know the cost of milk at any store ahead of time, so you need to efficiently contact each one and determine the minimum price to know whether it fits in your budget. If we were discussing this with a friend, we might say, “I need to find out how much milk costs at all the stores around here.” To solve that problem in a programming language, these words imply precise operations on sets of information. We can write the following “pseudocode,” which uses words that indicate logical thinking but that cannot be pasted directly into a program: +Suppose you need to go shopping for milk, and you have two criteria for determining where you will buy your milk: location and price. The store needs to be close to your home, and as a first step in deciding whether to buy milk today, you want to identify the lowest price among those stores. You don’t know the cost of milk at any store ahead of time, so you need to efficiently contact each one and determine the minimum price to know whether it fits in your budget. If we were discussing this with a friend, we might say, “I need to find out how much milk costs at all the stores around here.” To solve that problem in a programming language, these words imply precise operations on sets of information. We can write the following “pseudocode,” which uses words that indicate logical thinking but that cannot be pasted directly into a program: AllStoresOnEarth.filterNearbyStores.filterStoresWithMilk.getMilkPricesFromEachStore.determineTheMinimumValue -Imagine doing these actions not on a computer but in a more old-fashioned way: calling on the telephone for milk prices, writing the milk prices on paper, and inspecting the list to find the lowest value. In this approach, we begin with AllStoresOnEarth, since there is at least some possibility that we could decide to visit any store on Earth, a set that could include millions of stores, with prices for millions or billions of items. A wise first action would be to limit ourselves to nearby stores. Asking to filterNearbyStores would reduce the number of potential stores to hundreds, depending on how far we are willing to travel for milk. Then, working with that smaller set, we further filterStoresWithMilk, limiting ourselves to stores that sell our target item. At that point in the filtering, imagine that just 10 possibilities remain. Then, by telephone, we getMilkPricesFromEachStore, making a short paper list of prices. We then scan the list to determineTheMinimumValue to decide which store to visit. +Imagine doing these actions not on a computer but in a more old-fashioned way: calling on the telephone for milk prices, writing the milk prices on paper, and inspecting the list to find the lowest value. In this approach, we begin with AllStoresOnEarth, since there is at least some possibility that we could decide to visit any store on Earth, a set that could include millions of stores, with prices for millions or billions of items. A wise first action would be to limit ourselves to nearby stores. Asking to filterNearbyStores would reduce the number of potential stores to hundreds, depending on how far we are willing to travel for milk. Then, working with that smaller set, we further filterStoresWithMilk, limiting ourselves to stores that sell our target item. At that point in the filtering, imagine that just 10 possibilities remain. Then, by telephone, we getMilkPricesFromEachStore, making a short paper list of prices. We then scan the list to determineTheMinimumValue to decide which store to visit. -In that example, each color plays a different role in the workflow. The AllStoresOnEarth set, any one of which might contain inexpensive milk, is an enormous collection. The filtering actions filterNearbyStores and filterStoresWithMilk are operations that can happen on any set of stores. These actions take a set of stores, do some operation to limit that set, and return that smaller set of stores as an answer. The action to getMilkPricesFromEachStore takes a simple idea—calling a store for a milk price—and “maps” it over a given set of stores. Finally, with the list of nearby milk prices assembled, the action to determineTheMinimumValue, a general idea that could be applied to any list of numbers, identifies the cheapest one. +In that example, each color plays a different role in the workflow. The AllStoresOnEarth set, any one of which might contain inexpensive milk, is an enormous collection. The filtering actions filterNearbyStores and filterStoresWithMilk are operations that can happen on any set of stores. These actions take a set of stores, do some operation to limit that set, and return that smaller set of stores as an answer. The action to getMilkPricesFromEachStore takes a simple idea—calling a store for a milk price—and “maps” it over a given set of stores. Finally, with the list of nearby milk prices assembled, the action to determineTheMinimumValue, a general idea that could be applied to any list of numbers, identifies the cheapest one. -The list of steps above might seem almost too obvious, but the choice and order of operations can have a big impact on the feasibility of the problem. Imagine if we had decided to do the same operations in a slightly different order: +The list of steps above might seem almost too obvious, but the choice and order of operations can have a big impact on the feasibility of the problem. Imagine if we had decided to do the same operations in a slightly different order: AllStoresOnEarth.filterStoresWithMilk.getMilkPricesFromEachStore.filterNearbyStores.determineMinimumValue -In this approach, we first identify all the stores on Earth that have milk, then contact them one by one to get their current milk price. If the contact is done by phone, this could be a painfully slow process involving millions of phone calls. It would take considerable “processing” time to make each call, and careful work to record each price onto a giant list. Processing the operations in this order would demand that only after entirely finishing the process of contacting every milk proprietor on Earth, we then identify the ones on our list that are not nearby enough to visit, then scan the prices on the list of nearby stores to find the cheapest one. This should ultimately give the same answer as the more efficient first example, but only after requiring so much effort that we might want to give up. +In this approach, we first identify all the stores on Earth that have milk, then contact them one by one to get their current milk price. If the contact is done by phone, this could be a painfully slow process involving millions of phone calls. It would take considerable “processing” time to make each call, and careful work to record each price onto a giant list. Processing the operations in this order would demand that only after entirely finishing the process of contacting every milk proprietor on Earth, we then identify the ones on our list that are not nearby enough to visit, then scan the prices on the list of nearby stores to find the cheapest one. This should ultimately give the same answer as the more efficient first example, but only after requiring so much effort that we might want to give up. In addition to the greater order of magnitude of the list size, you can see that there are also possible slow points in the process. Could you make a million phone calls yourself? Maybe, but it might be pretty appealing to hire, say, 1000 people to help. While being able to make a large number of calls in parallel would speed up the calling stage, it’s important to note that you would need to wait for all 1000 callers to return their sublists of prices. Why wait? Nearby stores could be on any caller’s sublist, so any caller might be the one to find the lowest nearby price. The identification of the lowest nearby price would need to wait for the slowest caller, even if it turned out that all of that last caller’s prices came from stores on the other side of the world. @@ -70,166 +70,166 @@ This counterexample would also have other complications—such as the need to tr ## Filtering Image Collections in Earth Engine -The first part of the filter, map, reduce paradigm is “filtering” to get a smaller ImageCollection from a larger one. As in the milk example, filters take a large set of items, limit it by some criterion, and return a smaller set for consideration. Here, filters take an ImageCollection, limit it by some criterion of date, location, or image characteristics, and return a smaller ImageCollection (Fig. F4.0.1). +The first part of the filter, map, reduce paradigm is “filtering” to get a smaller ImageCollection from a larger one. As in the milk example, filters take a large set of items, limit it by some criterion, and return a smaller set for consideration. Here, filters take an ImageCollection, limit it by some criterion of date, location, or image characteristics, and return a smaller ImageCollection (Fig. F4.0.1). -![](F4/image66.png) +![Fig. 4.0.1 Filter, map, reduce as applied to image collections in Earth Engine](F4/image66.png) -Fig. 4.0.1 Filter, map, reduce as applied to image collections in Earth Engine -As described first in Chap. F1.2, the Earth Engine API provides a set of filters for the ImageCollection type. The filters can limit an ImageCollection based on spatial, temporal, or attribute characteristics. Filters were used in Parts F1, F2, and F3 without much context or explanation, to isolate an image from an ImageCollection for inspection or manipulation. The information below should give perspective on that work while introducing some new tools for filtering image collections. +As described first in Chap. F1.2, the Earth Engine API provides a set of filters for the ImageCollection type. The filters can limit an ImageCollection based on spatial, temporal, or attribute characteristics. Filters were used in Parts F1, F2, and F3 without much context or explanation, to isolate an image from an ImageCollection for inspection or manipulation. The information below should give perspective on that work while introducing some new tools for filtering image collections. -Below are three examples of limiting a Landsat 5 ImageCollection by characteristics and assessing the size of the resulting set. +Below are three examples of limiting a Landsat 5 ImageCollection by characteristics and assessing the size of the resulting set. -FilterDate This takes an ImageCollection as input and returns an ImageCollection whose members satisfy the specified date criteria. We’ll adapt the earlier filtering logic seen in Chap. F1.2: +FilterDate This takes an ImageCollection as input and returns an ImageCollection whose members satisfy the specified date criteria. We’ll adapt the earlier filtering logic seen in Chap. F1.2: -var imgCol = ee.ImageCollection('LANDSAT/LT05/C02/T1_L2'); +var imgCol = ee.ImageCollection('LANDSAT/LT05/C02/T1_L2'); +```js // How many Tier 1 Landsat 5 images have ever been collected? print("All images ever: ", imgCol.size()); // A very large number // How many images were collected in the 2000s? -var startDate = '2000-01-01'; -var endDate = '2010-01-01'; +var startDate = '2000-01-01'; +var endDate = '2010-01-01'; -var imgColfilteredByDate = imgCol.filterDate(startDate, endDate); +var imgColfilteredByDate = imgCol.filterDate(startDate, endDate); print("All images 2000-2010: ", imgColfilteredByDate.size()); // A smaller (but still large) number -After running the code, you should get a very large number for the full set of images. You also will likely get a very large number for the subset of images over the decade-scale interval.   +After running the code, you should get a very large number for the full set of images. You also will likely get a very large number for the subset of images over the decade-scale interval. -FilterBounds It may be that—similar to the milk example—only images near to a place of interest are useful for you. As first presented in Part F1, filterBounds takes an ImageCollection as input and returns an ImageCollection whose images surround a specified location. If we take the ImageCollection that was filtered by date and then filter it by bounds, we will have filtered the collection to those images near a specified point within the specified date interval. With the code below, we’ll count the number of images in the Shanghai vicinity, first visited in Chap. F1.1, from the early 2000s: +FilterBounds It may be that—similar to the milk example—only images near to a place of interest are useful for you. As first presented in Part F1, filterBounds takes an ImageCollection as input and returns an ImageCollection whose images surround a specified location. If we take the ImageCollection that was filtered by date and then filter it by bounds, we will have filtered the collection to those images near a specified point within the specified date interval. With the code below, we’ll count the number of images in the Shanghai vicinity, first visited in Chap. F1.1, from the early 2000s: -var ShanghaiImage = ee.Image(   'LANDSAT/LT05/C02/T1_L2/LT05_118038_20000606'); +var ShanghaiImage = ee.Image( 'LANDSAT/LT05/C02/T1_L2/LT05_118038_20000606'); Map.centerObject(ShanghaiImage, 9); -var imgColfilteredByDateHere = imgColfilteredByDate.filterBounds(Map   .getCenter()); +var imgColfilteredByDateHere = imgColfilteredByDate.filterBounds(Map .getCenter()); print("All images here, 2000-2010: ", imgColfilteredByDateHere .size()); // A smaller number -If you’d like, you could take a few minutes to explore the behavior of the script in different parts of the world. To do that, you would need to comment out the Map.centerObject command to keep the map from moving to that location each time you run the script. +If you’d like, you could take a few minutes to explore the behavior of the script in different parts of the world. To do that, you would need to comment out the Map.centerObject command to keep the map from moving to that location each time you run the script. -Filter by Other Image Metadata        As first explained in Chap. F1.3, the date and location of an image are characteristics stored with each image. Another important factor in image processing is the cloud cover, an image-level value computed for each image in many collections, including the Landsat and Sentinel-2 collections. The overall cloudiness score might be stored under different metadata tag names in different data sets. For example, for Sentinel-2, this overall cloudiness score is stored in the CLOUDY_PIXEL_PERCENTAGE metadata field. For Landsat 5, the ImageCollection we are using in this example, the image-level cloudiness score is stored using the tag CLOUD_COVER. If you are unfamiliar with how to find this information, these skills are first presented in Part F1. +Filter by Other Image Metadata As first explained in Chap. F1.3, the date and location of an image are characteristics stored with each image. Another important factor in image processing is the cloud cover, an image-level value computed for each image in many collections, including the Landsat and Sentinel-2 collections. The overall cloudiness score might be stored under different metadata tag names in different data sets. For example, for Sentinel-2, this overall cloudiness score is stored in the CLOUDY_PIXEL_PERCENTAGE metadata field. For Landsat 5, the ImageCollection we are using in this example, the image-level cloudiness score is stored using the tag CLOUD_COVER. If you are unfamiliar with how to find this information, these skills are first presented in Part F1. -Here, we will access the ImageCollection that we just built using filterBounds and filterDate, and then further filter the images by the image-level cloud cover score, using the filterMetadata function. +Here, we will access the ImageCollection that we just built using filterBounds and filterDate, and then further filter the images by the image-level cloud cover score, using the filterMetadata function. -Next, let’s remove any images with 50% or more cloudiness. As will be described in subsequent chapters working with per-pixel cloudiness information, you might want to retain those images in a real-life study, if you feel some values within cloudy images might be useful. For now, to illustrate the filtering concept, let’s keep only images whose image-level cloudiness values indicate that the cloud coverage is lower than 50%. Here, we will take the set already filtered by bounds and date, and further filter it using the cloud percentage into a new ImageCollection. Add this line to the script to filter by cloudiness and print the size to the Console. +Next, let’s remove any images with 50% or more cloudiness. As will be described in subsequent chapters working with per-pixel cloudiness information, you might want to retain those images in a real-life study, if you feel some values within cloudy images might be useful. For now, to illustrate the filtering concept, let’s keep only images whose image-level cloudiness values indicate that the cloud coverage is lower than 50%. Here, we will take the set already filtered by bounds and date, and further filter it using the cloud percentage into a new ImageCollection. Add this line to the script to filter by cloudiness and print the size to the Console. -var L5FilteredLowCloudImages = imgColfilteredByDateHere -   .filterMetadata('CLOUD_COVER', 'less_than', 50); +var L5FilteredLowCloudImages = imgColfilteredByDateHere + .filterMetadata('CLOUD_COVER', 'less_than', 50); print("Less than 50% clouds in this area, 2000-2010", -   L5FilteredLowCloudImages.size()); // A smaller number + L5FilteredLowCloudImages.size()); // A smaller number -Filtering in an Efficient Order As you saw earlier in the hypothetical milk example, we typically filter, then map, and then reduce, in that order. In the same way that we would not want to call every store on Earth, preferring instead to narrow down the list of potential stores first, we filter images first in our workflow in Earth Engine. In addition, you may have noticed that the ordering of the filters within the filtering stage also mattered in the milk example. This is also true in Earth Engine. For problems with a non-global spatial component in which filterBounds is to be used, it is most efficient to do that spatial filtering first. +Filtering in an Efficient Order As you saw earlier in the hypothetical milk example, we typically filter, then map, and then reduce, in that order. In the same way that we would not want to call every store on Earth, preferring instead to narrow down the list of potential stores first, we filter images first in our workflow in Earth Engine. In addition, you may have noticed that the ordering of the filters within the filtering stage also mattered in the milk example. This is also true in Earth Engine. For problems with a non-global spatial component in which filterBounds is to be used, it is most efficient to do that spatial filtering first. -In the code below, you will see that you can “chain” the filter commands, which are then executed from left to right. Below, we chain the filters in the same order as you specified above. Note that it gives an ImageCollection of the same size as when you applied the filters one at a time. +In the code below, you will see that you can “chain” the filter commands, which are then executed from left to right. Below, we chain the filters in the same order as you specified above. Note that it gives an ImageCollection of the same size as when you applied the filters one at a time. -var chainedFilteredSet = imgCol.filterDate(startDate, endDate) -   .filterBounds(Map.getCenter()) -   .filterMetadata('CLOUD_COVER', 'less_than', 50); +var chainedFilteredSet = imgCol.filterDate(startDate, endDate) + .filterBounds(Map.getCenter()) + .filterMetadata('CLOUD_COVER', 'less_than', 50); print('Chained: Less than 50% clouds in this area, 2000-2010', -   chainedFilteredSet.size()); + chainedFilteredSet.size()); -In the code below, we chain the filters in a more efficient order, implementing filterBounds first. This, too, gives an ImageCollection of the same size as when you applied the filters in the less efficient order, whether the filters were chained or not. +``` +In the code below, we chain the filters in a more efficient order, implementing filterBounds first. This, too, gives an ImageCollection of the same size as when you applied the filters in the less efficient order, whether the filters were chained or not. -var efficientFilteredSet = imgCol.filterBounds(Map.getCenter()) -   .filterDate(startDate, endDate) -   .filterMetadata('CLOUD_COVER', 'less_than', 50); +var efficientFilteredSet = imgCol.filterBounds(Map.getCenter()) + .filterDate(startDate, endDate) + .filterMetadata('CLOUD_COVER', 'less_than', 50); print('Efficient filtering: Less than 50% clouds in this area, 2000-2010', -   efficientFilteredSet.size()); + efficientFilteredSet.size()); -Each of the two chained sets of operations will give the same result as before for the number of images. While the second order is more efficient, both approaches are likely to return the answer to the Code Editor at roughly the same time for this very small example. The order of operations is most important in larger problems in which you might be challenged to manage memory carefully. As in the milk example in which you narrowed geographically first, it is good practice in Earth Engine to order the filters with the filterBounds first, followed by metadata filters in order of decreasing specificity. +Each of the two chained sets of operations will give the same result as before for the number of images. While the second order is more efficient, both approaches are likely to return the answer to the Code Editor at roughly the same time for this very small example. The order of operations is most important in larger problems in which you might be challenged to manage memory carefully. As in the milk example in which you narrowed geographically first, it is good practice in Earth Engine to order the filters with the filterBounds first, followed by metadata filters in order of decreasing specificity. -::: {.callout-note} -Code Checkpoint F40a. The book’s repository contains a script that shows what your code should look like at this point. +:::{.callout-note} +Code Checkpoint F40a. The book’s repository contains a script that shows what your code should look like at this point. ::: Now, with an efficiently filtered collection that satisfies our chosen criteria, we will next explore the second stage: executing a function for all of the images in the set. ## Mapping over Image Collections in Earth Engine -In Chap. F3.1, we calculated the Enhanced Vegetation Index (EVI) in very small steps to illustrate band arithmetic on satellite images. In that chapter, code was called once, on a single image. What if we wanted to compute the EVI in the same way for every image of an entire ImageCollection? Here, we use the key tool for the second part of the workflow in Earth Engine, a .map command (Fig. F4.0.1). This is roughly analogous to the step of making phone calls in the milk example that began this chapter, in which you took a list of store names and transformed it through effort into a list of milk prices. +In Chap. F3.1, we calculated the Enhanced Vegetation Index (EVI) in very small steps to illustrate band arithmetic on satellite images. In that chapter, code was called once, on a single image. What if we wanted to compute the EVI in the same way for every image of an entire ImageCollection? Here, we use the key tool for the second part of the workflow in Earth Engine, a .map command (Fig. F4.0.1). This is roughly analogous to the step of making phone calls in the milk example that began this chapter, in which you took a list of store names and transformed it through effort into a list of milk prices. -Before beginning to code the EVI functionality, it’s worth noting that the word “map” is encountered in multiple settings during cloud-based remote sensing, and it’s important to be able to distinguish the uses. A good way to think of it is that “map” can act as a verb or as a noun in Earth Engine. There are two uses of “map” as a noun. We might refer casually to “the map,” or more precisely to “the  Map panel”; these terms refer to the place where the images are shown in the code interface. A second way “map” is used as a noun is to refer to an Earth Engine object, which has functions that can be called on it. Examples of this are the familiar Map.addLayer and Map.setCenter. Where that use of the word is intended, it will be shown in purple text and capitalized in the Code Editor. What we are discussing here is the use of .map as a verb, representing the idea of performing a set of actions repeatedly on a set. This is typically referred to as “mapping over the set.”   +Before beginning to code the EVI functionality, it’s worth noting that the word “map” is encountered in multiple settings during cloud-based remote sensing, and it’s important to be able to distinguish the uses. A good way to think of it is that “map” can act as a verb or as a noun in Earth Engine. There are two uses of “map” as a noun. We might refer casually to “the map,” or more precisely to “the Map panel”; these terms refer to the place where the images are shown in the code interface. A second way “map” is used as a noun is to refer to an Earth Engine object, which has functions that can be called on it. Examples of this are the familiar Map.addLayer and Map.setCenter. Where that use of the word is intended, it will be shown in purple text and capitalized in the Code Editor. What we are discussing here is the use of .map as a verb, representing the idea of performing a set of actions repeatedly on a set. This is typically referred to as “mapping over the set.” -To map a given set of operations efficiently over an entire ImageCollection, the processing needs to be set up in a particular way. Users familiar with other programming languages might expect to see “loop” code to do this, but the processing is not done exactly that way in Earth Engine. Instead, we will create a function, and then map it over the ImageCollection. To begin, envision creating a function that takes exactly one parameter, an ee.Image. The function is then designed to perform a specified set of operations on the input ee.Image and then, importantly, returns an ee.Image as the last step of the function. When we map that function over an ImageCollection, as we’ll illustrate below, the effect is that we begin with an ImageCollection, do operations to each image, and receive a processed ImageCollection as the output. +To map a given set of operations efficiently over an entire ImageCollection, the processing needs to be set up in a particular way. Users familiar with other programming languages might expect to see “loop” code to do this, but the processing is not done exactly that way in Earth Engine. Instead, we will create a function, and then map it over the ImageCollection. To begin, envision creating a function that takes exactly one parameter, an ee.Image. The function is then designed to perform a specified set of operations on the input ee.Image and then, importantly, returns an ee.Image as the last step of the function. When we map that function over an ImageCollection, as we’ll illustrate below, the effect is that we begin with an ImageCollection, do operations to each image, and receive a processed ImageCollection as the output. -What kinds of functions could we create? For example, you could imagine a function taking an image and returning an image whose pixels have the value 1 where the value of a given band was lower than a certain threshold, and 0 otherwise. The effect of mapping this function would be an entire ImageCollection of images with zeroes and ones representing the results of that test on each image. Or you could imagine a function computing a complex self-defined index and sending back an image of that index calculated in each pixel. Here, we’ll create a function to compute the EVI for any input Landsat 5 image and return the one-band image for which the index is computed for each pixel. Copy and paste the function definition below into the Code Editor, adding it to the end of the script from the previous section. +What kinds of functions could we create? For example, you could imagine a function taking an image and returning an image whose pixels have the value 1 where the value of a given band was lower than a certain threshold, and 0 otherwise. The effect of mapping this function would be an entire ImageCollection of images with zeroes and ones representing the results of that test on each image. Or you could imagine a function computing a complex self-defined index and sending back an image of that index calculated in each pixel. Here, we’ll create a function to compute the EVI for any input Landsat 5 image and return the one-band image for which the index is computed for each pixel. Copy and paste the function definition below into the Code Editor, adding it to the end of the script from the previous section. -var makeLandsat5EVI = function(oneL5Image) {   // compute the EVI for any Landsat 5 image. Note it's specific to   // Landsat 5 images due to the band numbers. Don't run this exact   // function for images from sensors other than Landsat 5.   // Extract the bands and divide by 1e4 to account for scaling done.   var nirScaled = oneL5Image.select('SRvide(10000);   var redScaled = oneL5Image.select('SR_B3').divide(10000);   var blueScaled = oneL5Image.select('SR_B1').divide(10000);   // Calculate the numerator, note that order goes from left to right.   var numeratorEVI = (nirScaled.subtract(redScaled)).multiply(       2.5);   // Calculate the denominator   var denomClause1 = redScaled.multiply(6);   var denomClause2 = blueScaled.multiply(7.5);   var denominatorEVI = nirScaled.add(denomClause1).subtract( -       denomClause2).add(1);   // Calculate EVI and name it.   var landsat5EVI = numeratorEVI.divide(denominatorEVI).rename(       'EVI');   return (landsat5EVI); +var makeLandsat5EVI = function(oneL5Image) { // compute the EVI for any Landsat 5 image. Note it's specific to // Landsat 5 images due to the band numbers. Don't run this exact // function for images from sensors other than Landsat 5. // Extract the bands and divide by 1e4 to account for scaling done. var nirScaled = oneL5Image.select('SRvide(10000); var redScaled = oneL5Image.select('SR_B3').divide(10000); var blueScaled = oneL5Image.select('SR_B1').divide(10000); // Calculate the numerator, note that order goes from left to right. var numeratorEVI = (nirScaled.subtract(redScaled)).multiply( 2.5); // Calculate the denominator var denomClause1 = redScaled.multiply(6); var denomClause2 = blueScaled.multiply(7.5); var denominatorEVI = nirScaled.add(denomClause1).subtract( + denomClause2).add(1); // Calculate EVI and name it. var landsat5EVI = numeratorEVI.divide(denominatorEVI).rename( 'EVI'); return (landsat5EVI); }; -It is worth emphasizing that, in general, band names are specific to each ImageCollection. As a result, if that function were run on an image without the band 'SR_B4', for example, the function call would fail. Here, we have emphasized in the function’s name that it is specifically for creating EVI for Landsat 5.   +It is worth emphasizing that, in general, band names are specific to each ImageCollection. As a result, if that function were run on an image without the band 'SR_B4', for example, the function call would fail. Here, we have emphasized in the function’s name that it is specifically for creating EVI for Landsat 5. -The function makeLandsat5EVI is built to receive a single image, select the proper bands for calculating EVI, make the calculation, and return a one-banded image. If we had the name of each image comprising our ImageCollection, we could enter the names into the Code Editor and call the function one at a time for each, assembling the images into variables, and then combining them into an ImageCollection. This would be very tedious and highly prone to mistakes: lists of items might get mistyped, an image might be missed, etc. Instead, as mentioned above, we will use .map. With the code below, let’s print the information about the cloud-filtered collection and display it, execute the .map command, and explore the resulting ImageCollection. +The function makeLandsat5EVI is built to receive a single image, select the proper bands for calculating EVI, make the calculation, and return a one-banded image. If we had the name of each image comprising our ImageCollection, we could enter the names into the Code Editor and call the function one at a time for each, assembling the images into variables, and then combining them into an ImageCollection. This would be very tedious and highly prone to mistakes: lists of items might get mistyped, an image might be missed, etc. Instead, as mentioned above, we will use .map. With the code below, let’s print the information about the cloud-filtered collection and display it, execute the .map command, and explore the resulting ImageCollection. -var L5EVIimages = efficientFilteredSet.map(makeLandsat5EVI); +var L5EVIimages = efficientFilteredSet.map(makeLandsat5EVI); print('Verifying that the .map gives back the same number of images: ', -   L5EVIimages.size()); + L5EVIimages.size()); print(L5EVIimages); Map.addLayer(L5EVIimages, {}, 'L5EVIimages', 1, 1); -After entering and executing this code, you will see a grayscale image. If you look closely at the edges of the image, you might spot other images drawn behind it in a way that looks somewhat like a stack of papers on a table. This is the drawing of the ImageCollection made from the makeLandsat5EVI function. You can select the Inspector panel and click on one of the grayscale pixels to view the values of the entire ImageCollection. After clicking on a pixel, look for the Series tag by opening and closing the list of items. When you open that tag, you will see a chart of the EVI values at that pixel, created by mapping the makeLandsat5EVI function over the filtered ImageCollection. +After entering and executing this code, you will see a grayscale image. If you look closely at the edges of the image, you might spot other images drawn behind it in a way that looks somewhat like a stack of papers on a table. This is the drawing of the ImageCollection made from the makeLandsat5EVI function. You can select the Inspector panel and click on one of the grayscale pixels to view the values of the entire ImageCollection. After clicking on a pixel, look for the Series tag by opening and closing the list of items. When you open that tag, you will see a chart of the EVI values at that pixel, created by mapping the makeLandsat5EVI function over the filtered ImageCollection. -::: {.callout-note} -Code Checkpoint F40b. The book’s repository contains a script that shows what your code should look like at this point. +:::{.callout-note} +Code Checkpoint F40b. The book’s repository contains a script that shows what your code should look like at this point. ::: ## Reducing an Image Collection -The third part of the filter, map, reduce paradigm is “reducing” values in an ImageCollection to extract meaningful values (Fig. F4.0.1). In the milk example, we reduced a large list of milk prices to find the minimum value. The Earth Engine API provides a large set of reducers for reducing a set of values to a summary statistic. +The third part of the filter, map, reduce paradigm is “reducing” values in an ImageCollection to extract meaningful values (Fig. F4.0.1). In the milk example, we reduced a large list of milk prices to find the minimum value. The Earth Engine API provides a large set of reducers for reducing a set of values to a summary statistic. -Here, you can think of each location, after the calculation of EVI has been executed though the .map command, as having a list of EVI values on it. Each pixel contains a potentially very large set of EVI values; the stack might be 15 items high in one location and perhaps 200, 2000, or 200,000 items high in another location, especially if a looser set of filters had been used. +Here, you can think of each location, after the calculation of EVI has been executed though the .map command, as having a list of EVI values on it. Each pixel contains a potentially very large set of EVI values; the stack might be 15 items high in one location and perhaps 200, 2000, or 200,000 items high in another location, especially if a looser set of filters had been used. -The code below computes the mean value, at every pixel, of the ImageCollection L5EVIimages created above. Add it at the bottom of your code. +The code below computes the mean value, at every pixel, of the ImageCollection L5EVIimages created above. Add it at the bottom of your code. -var L5EVImean = L5EVIimages.reduce(ee.Reducer.mean()); +var L5EVImean = L5EVIimages.reduce(ee.Reducer.mean()); print(L5EVImean); Map.addLayer(L5EVImean, { -   min: -1, -   max: 2, -   palette: ['red', 'white', 'green'] + min: -1, + max: 2, + palette: ['red', 'white', 'green'] }, 'Mean EVI'); -Using the same principle, the code below computes and draws the median value of the ImageCollection in every pixel. +Using the same principle, the code below computes and draws the median value of the ImageCollection in every pixel. -var L5EVImedian = L5EVIimages.reduce(ee.Reducer.median()); +var L5EVImedian = L5EVIimages.reduce(ee.Reducer.median()); print(L5EVImedian); Map.addLayer(L5EVImedian, { -   min: -1, -   max: 2, -   palette: ['red', 'white', 'green'] + min: -1, + max: 2, + palette: ['red', 'white', 'green'] }, 'Median EVI'); ![](F4/image43.png) -![](F4/image67.png) +![Fig. 4.0.2 The effects of two reducers on mapped EVI values in a filtered ImageCollection: mean image (above), and median image (below)](F4/image67.png) -Fig. 4.0.2 The effects of two reducers on mapped EVI values in a filtered ImageCollection: mean image (above), and median image (below) -There are many more reducers that work with an ImageCollection to produce a wide range of summary statistics. Reducers are not limited to returning only one item from the reduction. The minMax reducer, for example, returns a two-band image for each band it is given, one for the minimum and one for the maximum. +There are many more reducers that work with an ImageCollection to produce a wide range of summary statistics. Reducers are not limited to returning only one item from the reduction. The minMax reducer, for example, returns a two-band image for each band it is given, one for the minimum and one for the maximum. The reducers described here treat each pixel independently. In subsequent chapters in Part F4, you will see other kinds of reducers—for example, ones that summarize the characteristics in the neighborhood surrounding each pixel. -::: {.callout-note} -Code Checkpoint F40c. The book’s repository contains a script that shows what your code should look like at this point. +:::{.callout-note} +Code Checkpoint F40c. The book’s repository contains a script that shows what your code should look like at this point. ::: ## Synthesis {.unnumbered} -Assignment 1. Compare the mean and median images produced in Sect. 3 (Fig. 4.0.2). In what ways do they look different, and in what ways do they look alike? To understand how they work, pick a pixel and inspect the EVI values computed. In your opinion, which is a better representative of the data set? +Assignment 1. Compare the mean and median images produced in Sect. 3 (Fig. 4.0.2). In what ways do they look different, and in what ways do they look alike? To understand how they work, pick a pixel and inspect the EVI values computed. In your opinion, which is a better representative of the data set? Assignment 2. Adjust the filters to filter a different proportion of clouds, or a different date range. What effects do these changes have on the number of images and the look of the reductions made from them? -Assignment 3. Explore the ee.Filter options in the API documentation, and select a different filter that might be of interest. Filter images using it, and comment on the number of images and the reductions made from them. +Assignment 3. Explore the ee.Filter options in the API documentation, and select a different filter that might be of interest. Filter images using it, and comment on the number of images and the reductions made from them. -Assignment 4. Change the EVI function so that it returns the original image with the EVI band appended by replacing the return statement with this: return (oneL5Image.addBands(landsat5EVI)) +Assignment 4. Change the EVI function so that it returns the original image with the EVI band appended by replacing the return statement with this: return (oneL5Image.addBands(landsat5EVI)) -What does the median reducer return in that case? Some EVI values are 0. What are the conditions in which this occurs?   +What does the median reducer return in that case? Some EVI values are 0. What are the conditions in which this occurs? -Assignment 5. Choose a date and location that is important to you (e.g., your birthday and your place of birth). Filter Landsat imagery to get all the low-cloud imagery at your location within 6 months of the date. Then, reduce the ImageCollection to find the median EVI. Describe the image and how representative of the full range of values it is, in your opinion. +Assignment 5. Choose a date and location that is important to you (e.g., your birthday and your place of birth). Filter Landsat imagery to get all the low-cloud imagery at your location within 6 months of the date. Then, reduce the ImageCollection to find the median EVI. Describe the image and how representative of the full range of values it is, in your opinion. ## Conclusion {.unnumbered} -In this chapter, you learned about the paradigm of filter, map, reduce. You learned how to use these tools to sift through, operate on, and summarize a large set of images to suit your purposes. Using the Filter functionality, you learned how to take a large ImageCollection and filter away images that do not meet your criteria, retaining only those images that match a given set of characteristics. Using the Map functionality, you learned how to apply a function to each image in an ImageCollection, treating each image one at a time and executing a requested set of operations on each. Using the Reduce functionality, you learned how to summarize the elements of an ImageCollection, extracting summary values of interest. In the subsequent chapters of Part 4, you will encounter these concepts repeatedly, manipulating image collections according to your project needs using the building blocks seen here. By building on what you have done in this chapter, you will grow in your ability to do sophisticated projects in Earth Engine. +In this chapter, you learned about the paradigm of filter, map, reduce. You learned how to use these tools to sift through, operate on, and summarize a large set of images to suit your purposes. Using the Filter functionality, you learned how to take a large ImageCollection and filter away images that do not meet your criteria, retaining only those images that match a given set of characteristics. Using the Map functionality, you learned how to apply a function to each image in an ImageCollection, treating each image one at a time and executing a requested set of operations on each. Using the Reduce functionality, you learned how to summarize the elements of an ImageCollection, extracting summary values of interest. In the subsequent chapters of Part 4, you will encounter these concepts repeatedly, manipulating image collections according to your project needs using the building blocks seen here. By building on what you have done in this chapter, you will grow in your ability to do sophisticated projects in Earth Engine. @@ -239,7 +239,7 @@ In this chapter, you learned about the paradigm of filter, map, reduce. You lea -::: {.callout-tip} +:::{.callout-tip} # Chapter Information ## Author {.unlisted .unnumbered} @@ -256,10 +256,10 @@ Gennadii Donchyts This chapter teaches how to explore image collections, including their spatiotemporal extent, resolution, and values stored in images and image properties. You will learn how to map and inspect image collections using maps, charts, and interactive tools, and how to compute different statistics of values stored in image collections using reducers. ## Learning Outcomes {.unlisted .unnumbered} -  + -* Inspecting the spatiotemporal extent and resolution of image collections by mapping image geometry and plotting image time properties. -* Exploring properties of images stored in an ImageCollection by plotting charts and deriving statistics. +* Inspecting the spatiotemporal extent and resolution of image collections by mapping image geometry and plotting image time properties. +* Exploring properties of images stored in an ImageCollection by plotting charts and deriving statistics. * Filtering image collections by using stored or computed image properties. * Exploring the distribution of values stored in image pixels of an ImageCollection through percentile reducers. @@ -268,155 +268,165 @@ This chapter teaches how to explore image collections, including their spatiotem * Import images and image collections, filter, and visualize (Part F1). * Perform basic image analysis: select bands, compute indices, create masks (Part F2). -* Summarize an ImageCollection with reducers (Chap. F4.0). +* Summarize an ImageCollection with reducers (Chap. F4.0). *** -In the previous chapter (Chap. F4.0), the filter, map, reduce paradigm was introduced. The main goal of this chapter is to demonstrate some of the ways that those concepts can be used within Earth Engine to better understand the variability of values stored in image collections. Sect. 1 demonstrates how time-dependent values stored in the images of an ImageCollection can be inspected using the Code Editor user interface after filtering them to a limited spatiotemporal range (i.e., geometry and time ranges). Sect. 2 shows how the extent of images, as well as basic statistics, such as the number of observations, can be visualized to better understand the spatiotemporal extent of image collections. Then, Sects. 3 and 4 demonstrate how simple reducers such as mean and median, and more advanced reducers such as percentiles, can be used to better understand how the values of a filtered ImageCollection are distributed. +In the previous chapter (Chap. F4.0), the filter, map, reduce paradigm was introduced. The main goal of this chapter is to demonstrate some of the ways that those concepts can be used within Earth Engine to better understand the variability of values stored in image collections. Sect. 1 demonstrates how time-dependent values stored in the images of an ImageCollection can be inspected using the Code Editor user interface after filtering them to a limited spatiotemporal range (i.e., geometry and time ranges). Sect. 2 shows how the extent of images, as well as basic statistics, such as the number of observations, can be visualized to better understand the spatiotemporal extent of image collections. Then, Sects. 3 and 4 demonstrate how simple reducers such as mean and median, and more advanced reducers such as percentiles, can be used to better understand how the values of a filtered ImageCollection are distributed. ## Filtering and Inspecting an Image Collection -We will focus on the area in and surrounding Lisbon, Portugal. Below, we will define a point, lisbonPoint, located in the city; access the very large Landsat ImageCollection and limit it to the year 2020 and to the images that contain Lisbon; and select bands 6, 5, and 4 from each of the images in the resulting filtered ImageCollection. +We will focus on the area in and surrounding Lisbon, Portugal. Below, we will define a point, lisbonPoint, located in the city; access the very large Landsat ImageCollection and limit it to the year 2020 and to the images that contain Lisbon; and select bands 6, 5, and 4 from each of the images in the resulting filtered ImageCollection. +```js // Define a region of interest as a point in Lisbon, Portugal. -var lisbonPoint = ee.Geometry.Point(-9.179473, 38.763948); +var lisbonPoint = ee.Geometry.Point(-9.179473, 38.763948); // Center the map at that point. Map.centerObject(lisbonPoint, 16); // filter the large ImageCollection to be just images from 2020 // around Lisbon. From each image, select true-color bands to draw -var filteredIC = ee.ImageCollection('LANDSAT/LC08/C02/T1_TOA') -   .filterDate('2020-01-01', '2021-01-01') -   .filterBounds(lisbonPoint) -   .select(['B6', 'B5', 'B4']); +var filteredIC = ee.ImageCollection('LANDSAT/LC08/C02/T1_TOA') + .filterDate('2020-01-01', '2021-01-01') + .filterBounds(lisbonPoint) + .select(['B6', 'B5', 'B4']); // Add the filtered ImageCollection so that we can inspect values // via the Inspector tool Map.addLayer(filteredIC, {}, 'TOA image collection'); -The three selected bands (which correspond to SWIR1, NIR, and Red) display a false-color image that accentuates differences between different land covers (e.g., concrete, vegetation) in Lisbon. With the Inspector tab highlighted (Fig. F4.1.1), clicking on a point will bring up the values of bands 6, 5, and 4 from each of the images. If you open the Series option, you’ll see the values through time. For the specified point and for all other points in Lisbon (since they are all enclosed in the same Landsat scene), there are 16 images gathered in 2020. By following one of the graphed lines (in blue, yellow, or red) with your finger, you should be able to count that many distinct values. Moving the mouse along the lines will show the specific values and the image dates. +``` +The three selected bands (which correspond to SWIR1, NIR, and Red) display a false-color image that accentuates differences between different land covers (e.g., concrete, vegetation) in Lisbon. With the Inspector tab highlighted (Fig. F4.1.1), clicking on a point will bring up the values of bands 6, 5, and 4 from each of the images. If you open the Series option, you’ll see the values through time. For the specified point and for all other points in Lisbon (since they are all enclosed in the same Landsat scene), there are 16 images gathered in 2020. By following one of the graphed lines (in blue, yellow, or red) with your finger, you should be able to count that many distinct values. Moving the mouse along the lines will show the specific values and the image dates. -![](F4/image22.png) +![Fig. F4.1.1 Inspect values in an ImageCollection at a selected point by making use of the Inspector tool in the Code Editor](F4/image22.png) -Fig. F4.1.1 Inspect values in an ImageCollection at a selected point by making use of the Inspector tool in the Code Editor -We can also show this kind of chart automatically by making use of the ui.Chart function of the Earth Engine API. The following code snippet should result in the same chart as we could observe in the Inspector tab, assuming the same pixel is clicked. +We can also show this kind of chart automatically by making use of the ui.Chart function of the Earth Engine API. The following code snippet should result in the same chart as we could observe in the Inspector tab, assuming the same pixel is clicked. +```js // Construct a chart using values queried from image collection. -var chart = ui.Chart.image.series({ -   imageCollection: filteredIC, -   region: lisbonPoint, -   reducer: ee.Reducer.first(), -   scale: 10 +var chart = ui.Chart.image.series({ + imageCollection: filteredIC, + region: lisbonPoint, + reducer: ee.Reducer.first(), + scale: 10 }); // Show the chart in the Console. print(chart); -::: {.callout-note} -Code Checkpoint F41a. The book’s repository contains a script that shows what your code should look like at this point. +``` +:::{.callout-note} +Code Checkpoint F41a. The book’s repository contains a script that shows what your code should look like at this point. ::: -## How Many Images Are There, Everywhere on Earth? +## How Many Images Are There, Everywhere on Earth? -Suppose we are interested to find out how many valid observations we have at every map pixel on Earth for a given ImageCollection. This enormously computationally demanding task is surprisingly easy to do in Earth Engine. The API provides a set of reducer functions to summarize values to a single number in each pixel, as described in Chap. F4.0. We can apply this reducer, count, to our filtered ImageCollection with the code below. We’ll return to the same data set and filter for 2020, but without the geographic limitation. This will assemble images from all over the world, and then count the number of images in each pixel. The following code does that count, and adds the resulting image to the map with a predefined red/yellow/green color palette stretched between values 0 and 50. Continue pasting the code below into the same script. +Suppose we are interested to find out how many valid observations we have at every map pixel on Earth for a given ImageCollection. This enormously computationally demanding task is surprisingly easy to do in Earth Engine. The API provides a set of reducer functions to summarize values to a single number in each pixel, as described in Chap. F4.0. We can apply this reducer, count, to our filtered ImageCollection with the code below. We’ll return to the same data set and filter for 2020, but without the geographic limitation. This will assemble images from all over the world, and then count the number of images in each pixel. The following code does that count, and adds the resulting image to the map with a predefined red/yellow/green color palette stretched between values 0 and 50. Continue pasting the code below into the same script. +```js // compute and show the number of observations in an image collection -var count = ee.ImageCollection('LANDSAT/LC08/C02/T1_TOA') -   .filterDate('2020-01-01', '2021-01-01') -   .select(['B6']) -   .count(); +var count = ee.ImageCollection('LANDSAT/LC08/C02/T1_TOA') + .filterDate('2020-01-01', '2021-01-01') + .select(['B6']) + .count(); // add white background and switch to HYBRID basemap Map.addLayer(ee.Image(1), { -   palette: ['white'] + palette: ['white'] }, 'white', true, 0.5); Map.setOptions('HYBRID'); // show image count Map.addLayer(count, { -   min: 0, -   max: 50, -   palette: ['d7191c', 'fdae61', 'ffffbf', 'a6d96a',       '1a9641'] + min: 0, + max: 50, + palette: ['d7191c', 'fdae61', 'ffffbf', 'a6d96a', '1a9641'] }, 'landsat 8 image count (2020)'); // Center the map at that point. Map.centerObject(lisbonPoint, 5); +``` Run the command and zoom out. If the count of images over the entire Earth is viewed, the resulting map should look like Fig. F4.1.2. The created map data may take a few minutes to fully load in. -![](F4/image27.png) +![Fig. F4.1.2 The number of Landsat 8 images acquired during 2020](F4/image27.png) -Fig. F4.1.2 The number of Landsat 8 images acquired during 2020 Note the checkered pattern, somewhat reminiscent of a Mondrian painting. To understand why the image looks this way, it is useful to consider the overlapping image footprints. As Landsat passes over, each image is wide enough to produce substantial “sidelap” with the images from the adjacent paths, which are collected at different dates according to the satellite’s orbit schedule. In the north-south direction, there is also some overlap to ensure that there are no gaps in the data. Because these are served as distinct images and stored distinctly in Earth Engine, you will find that there can be two images from the same day with the same value for points in these overlap areas. Depending on the purposes of a study, you might find a way to ignore the duplicate pixel values during the analysis process. -You might have noticed that we summarized a single band from the original ImageCollection to ensure that the resulting image would give a single count in each pixel. The count reducer operates on every band passed to it. Since every image has the same number of bands, passing an ImageCollection of all seven Landsat bands to the count reducer would have returned seven identical values of 16 for every point. To limit any confusion from seeing the same number seven times, we selected one of the bands from each image in the collection. In your own work, you might want to use a different reducer, such as a median operation, that would give different, useful answers for each band. A few of these reducers are described below. +You might have noticed that we summarized a single band from the original ImageCollection to ensure that the resulting image would give a single count in each pixel. The count reducer operates on every band passed to it. Since every image has the same number of bands, passing an ImageCollection of all seven Landsat bands to the count reducer would have returned seven identical values of 16 for every point. To limit any confusion from seeing the same number seven times, we selected one of the bands from each image in the collection. In your own work, you might want to use a different reducer, such as a median operation, that would give different, useful answers for each band. A few of these reducers are described below. -::: {.callout-note} -Code Checkpoint F41b. The book’s repository contains a script that shows what your code should look like at this point. +:::{.callout-note} +Code Checkpoint F41b. The book’s repository contains a script that shows what your code should look like at this point. ::: -## Reducing Image Collections to Understand Band Values +## Reducing Image Collections to Understand Band Values -As we have seen, you could click at any point on Earth’s surface and see both the number of Landsat images recorded there in 2020 and the values of any image in any band through time. This is impressive and perhaps mind-bending, given the enormous amount of data in play. In this section and the next, we will explore two ways to summarize the numerical values of the bands—one straightforward way and one more complex but highly powerful way to see what information is contained in image collections. +As we have seen, you could click at any point on Earth’s surface and see both the number of Landsat images recorded there in 2020 and the values of any image in any band through time. This is impressive and perhaps mind-bending, given the enormous amount of data in play. In this section and the next, we will explore two ways to summarize the numerical values of the bands—one straightforward way and one more complex but highly powerful way to see what information is contained in image collections. -First, we will make a new layer that represents the mean value of each band in every pixel across every image from 2020 for the filtered set, add this layer to the layer set, and explore again with the Inspector. The previous section’s count reducer was called directly using a sort of simple shorthand; that could be done similarly here by calling mean on the assembled bands. In this example, we will use the reducer to get the mean using the more general reduce call. Continue pasting the code below into the same script. +First, we will make a new layer that represents the mean value of each band in every pixel across every image from 2020 for the filtered set, add this layer to the layer set, and explore again with the Inspector. The previous section’s count reducer was called directly using a sort of simple shorthand; that could be done similarly here by calling mean on the assembled bands. In this example, we will use the reducer to get the mean using the more general reduce call. Continue pasting the code below into the same script. +```js // Zoom to an informative scale for the code that follows. Map.centerObject(lisbonPoint, 10); // Add a mean composite image. -var meanFilteredIC = filteredIC.reduce(ee.Reducer.mean()); -Map.addLayer(meanFilteredIC, {},   'Mean values within image collection'); +var meanFilteredIC = filteredIC.reduce(ee.Reducer.mean()); +Map.addLayer(meanFilteredIC, {}, 'Mean values within image collection'); -Now, let’s look at the median value for each band among all the values gathered in 2020. Using the code below, calculate the median and explore the image with the Inspector. Compare this image briefly to the mean image by eye and by clicking in a few pixels in the Inspector. They should have different values, but in most places they will look very similar. +``` +Now, let’s look at the median value for each band among all the values gathered in 2020. Using the code below, calculate the median and explore the image with the Inspector. Compare this image briefly to the mean image by eye and by clicking in a few pixels in the Inspector. They should have different values, but in most places they will look very similar. +```js // Add a median composite image. -var medianFilteredIC = filteredIC.reduce(ee.Reducer.median()); -Map.addLayer(medianFilteredIC, {},   'Median values within image collection'); +var medianFilteredIC = filteredIC.reduce(ee.Reducer.median()); +Map.addLayer(medianFilteredIC, {}, 'Median values within image collection'); -There is a wide range of reducers available in Earth Engine. If you are curious about which reducers can be used to summarize band values across a collection of images, use the Docs tab in the Code Editor to list all reducers and look for those beginning with ee.Reducer. +``` +There is a wide range of reducers available in Earth Engine. If you are curious about which reducers can be used to summarize band values across a collection of images, use the Docs tab in the Code Editor to list all reducers and look for those beginning with ee.Reducer. -::: {.callout-note} +:::{.callout-note} Code Checkpoint F41c. The book’s repository contains a script that shows what your code should look like at this point. ::: ## Compute Multiple Percentile Images for an Image Collection -One particularly useful reducer that can help you better understand the variability of values in image collections is ee.Reducer.percentile. The nth percentile gives the value that is the nth largest in a set. In this context, you can imagine accessing all of the values for a given band in a given ImageCollection for a given pixel and sorting them. The 30th percentile, for example, is the value 30% of the way along the list from smallest to largest. This provides an easy way to explore the variability of the values in image collections by computing a cumulative density function of values on a per-pixel basis. The following code shows how we can calculate a single 30th percentile on a per-pixel and per-band basis for our Landsat 8 ImageCollection. Continue pasting the code below into the same script. +One particularly useful reducer that can help you better understand the variability of values in image collections is ee.Reducer.percentile. The nth percentile gives the value that is the nth largest in a set. In this context, you can imagine accessing all of the values for a given band in a given ImageCollection for a given pixel and sorting them. The 30th percentile, for example, is the value 30% of the way along the list from smallest to largest. This provides an easy way to explore the variability of the values in image collections by computing a cumulative density function of values on a per-pixel basis. The following code shows how we can calculate a single 30th percentile on a per-pixel and per-band basis for our Landsat 8 ImageCollection. Continue pasting the code below into the same script. +```js // compute a single 30% percentile -var p30 = filteredIC.reduce(ee.Reducer.percentile([30])); +var p30 = filteredIC.reduce(ee.Reducer.percentile([30])); Map.addLayer(p30, { -   min: 0.05, -   max: 0.35}, '30%'); + min: 0.05, + max: 0.35}, '30%'); -![](F4/image74.png) +``` +![Fig. F4.1.3 Landsat 8 TOA reflectance 30th percentile image computed for ImageCollection with images acquired during 2020 ](F4/image74.png) -Fig. F4.1.3 Landsat 8 TOA reflectance 30th percentile image computed for ImageCollection with images acquired during 2020  We can see that the resulting composite image (Fig. 4.1.3) has almost no cloudy pixels present for this area. This happens because cloudy pixels usually have higher reflectance values. At the lowest end of the values, other unwanted effects like cloud or hill shadows typically have very low reflectance values. This is why this 30th percentile composite image looks so much cleaner than the mean composite image (meanFilteredIC) calculated earlier. Note that the reducers operate per pixel: adjacent pixels are drawn from different images. This means that one pixel’s value could be taken from an image from one date, and the adjacent pixel’s value drawn from an entirely different period. Although, like the mean and median images, percentile images such as that seen in Fig. F4.1.3 never existed on a single day, composite images allow us to view Earth’s surface without the noise that can make analysis difficult. -We can explore the range of values in an entire ImageCollection by viewing a series of increasingly bright percentile images, as shown in Fig. F4.1.4. Paste and run the following code. +We can explore the range of values in an entire ImageCollection by viewing a series of increasingly bright percentile images, as shown in Fig. F4.1.4. Paste and run the following code. -var percentiles = [0, 10, 20, 30, 40, 50, 60, 70, 80]; +var percentiles = [0, 10, 20, 30, 40, 50, 60, 70, 80]; +```js // let's compute percentile images and add them as separate layers -percentiles.map(function(p) {   var image = filteredIC.reduce(ee.Reducer.percentile([p]));   Map.addLayer(image, { -       min: 0.05, -       max: 0.35   }, p + '%'); +percentiles.map(function(p) { var image = filteredIC.reduce(ee.Reducer.percentile([p])); Map.addLayer(image, { + min: 0.05, + max: 0.35 }, p + '%'); }); -Note that the code adds every percentile image as a separate map layer, so you need to go to the Layers control and show/hide different layers to explore differences. Here, we can see that low-percentile composite images depict darker, low-reflectance land features, such as water and cloud or hill shadows, while higher-percentile composite images (>70% in our example) depict clouds and any other atmospheric or land effects corresponding to bright reflectance values. +``` +Note that the code adds every percentile image as a separate map layer, so you need to go to the Layers control and show/hide different layers to explore differences. Here, we can see that low-percentile composite images depict darker, low-reflectance land features, such as water and cloud or hill shadows, while higher-percentile composite images (>70% in our example) depict clouds and any other atmospheric or land effects corresponding to bright reflectance values. -![](F4/image55.png) +![Fig. F4.1.4 Landsat 8 TOA reflectance percentile composite images](F4/image55.png) -Fig. F4.1.4 Landsat 8 TOA reflectance percentile composite images Earth Engine provides a very rich API, allowing users to explore image collections to better understand the extent and variability of data in space, time, and across bands, as well as tools to analyze values stored in image collections in a frequency domain. Exploring these values in different forms should be the first step of any study before developing data analysis algorithms. -::: {.callout-note} -Code Checkpoint F41d. The book’s repository contains a script that shows what your code should look like at this point. +:::{.callout-note} +Code Checkpoint F41d. The book’s repository contains a script that shows what your code should look like at this point. ::: ## Synthesis {.unnumbered} @@ -424,9 +434,9 @@ In the example above, the 30th percentile composite image would be useful for ty Assignment 1. Noting that your own interpretation of what constitutes a good composite is subjective, create a series of composites of a different location, or perhaps a pair of locations, for a given set of dates. -Assignment 2. Filter to create a relevant data set—for example, for Landsat 8 or Sentinel-2 over an agricultural growing season. Create percentile composites for a given location. Which image composite is the most satisfying, and what type of project do you have in mind when giving that response? +Assignment 2. Filter to create a relevant data set—for example, for Landsat 8 or Sentinel-2 over an agricultural growing season. Create percentile composites for a given location. Which image composite is the most satisfying, and what type of project do you have in mind when giving that response? -Assignment 3. Do you think it is possible to generalize about the relationship between the time window of an ImageCollection and the percentile value that will be the most useful for a given project, or will every region need to be inspected separately? +Assignment 3. Do you think it is possible to generalize about the relationship between the time window of an ImageCollection and the percentile value that will be the most useful for a given project, or will every region need to be inspected separately? ## Conclusion {.unnumbered} @@ -438,102 +448,101 @@ Wilson AM, Jetz W (2016) Remotely sensed high-resolution global cloud dynamics f -# Aggregating Images for Time Series +# Aggregating Images for Time Series -::: {.callout-tip} +:::{.callout-tip} # Chapter Information ## Author {.unlisted .unnumbered} -Ujaval Gandhi +Ujaval Gandhi ## Overview {.unlisted .unnumbered} -  + -Many remote sensing datasets consist of repeated observations over time. The interval between observations can vary widely. The Global Precipitation Measurement dataset, for example, produces observations of rain and snow worldwide every three hours. The Climate Hazards Group InfraRed Precipitation with Station (CHIRPS) project produces a gridded global dataset at the daily level and also for each five-day period. The Landsat 8 mission produces a new scene of each location on Earth every 16 days. With its constellation of two satellites, the Sentinel-2 mission images every location every five days. +Many remote sensing datasets consist of repeated observations over time. The interval between observations can vary widely. The Global Precipitation Measurement dataset, for example, produces observations of rain and snow worldwide every three hours. The Climate Hazards Group InfraRed Precipitation with Station (CHIRPS) project produces a gridded global dataset at the daily level and also for each five-day period. The Landsat 8 mission produces a new scene of each location on Earth every 16 days. With its constellation of two satellites, the Sentinel-2 mission images every location every five days. -Many applications, however, require computing aggregations of data at time intervals different from those at which the datasets were produced. For example, for determining rainfall anomalies, it is useful to compare monthly rainfall against a long-period monthly average. +Many applications, however, require computing aggregations of data at time intervals different from those at which the datasets were produced. For example, for determining rainfall anomalies, it is useful to compare monthly rainfall against a long-period monthly average. -While individual scenes are informative, many days are cloudy, and it is useful to build a robust cloud-free time series for many applications. Producing less cloudy or even cloud-free composites can be done by aggregating data to form monthly, seasonal, or yearly composites built from individual scenes. For example, if you are interested in detecting long-term changes in an urban landscape, creating yearly median composites can enable you to detect change patterns across long time intervals with less worry about day-to-day noise. +While individual scenes are informative, many days are cloudy, and it is useful to build a robust cloud-free time series for many applications. Producing less cloudy or even cloud-free composites can be done by aggregating data to form monthly, seasonal, or yearly composites built from individual scenes. For example, if you are interested in detecting long-term changes in an urban landscape, creating yearly median composites can enable you to detect change patterns across long time intervals with less worry about day-to-day noise. -This chapter will cover the techniques for aggregating individual images from a time  series at a chosen interval. We will take the CHIRPS time series of rainfall for one year and aggregate it to create a monthly rainfall time series. +This chapter will cover the techniques for aggregating individual images from a time series at a chosen interval. We will take the CHIRPS time series of rainfall for one year and aggregate it to create a monthly rainfall time series. ## Learning Outcomes {.unlisted .unnumbered} -* Using the Earth Engine API to work with dates. -* Aggregating values from an ImageCollection to calculate monthly, seasonal, or yearly images. +* Using the Earth Engine API to work with dates. +* Aggregating values from an ImageCollection to calculate monthly, seasonal, or yearly images. * Plotting the aggregated time series at a given location. ## Assumes you know how to:{.unlisted .unnumbered} * Import images and image collections, filter, and visualize (Part F1). -* ​​Create a graph using ui.Chart (Chap. F1.3). -* Write a function and map it over an ImageCollection (Chap. F4.0). -* Summarize an ImageCollection with reducers (Chap. F4.0, Chap. F4.1). +* ​​Create a graph using ui.Chart (Chap. F1.3). +* Write a function and map it over an ImageCollection (Chap. F4.0). +* Summarize an ImageCollection with reducers (Chap. F4.0, Chap. F4.1). * Inspect an Image and an ImageCollection, as well as their properties (Chap. F4.1). -::: +::: ## Introduction {.unlisted .unnumbered} -CHIRPS is a high-resolution global gridded rainfall dataset that combines satellite-measured precipitation with ground station data in a consistent, long time-series dataset. The data are provided by the University of California, Santa Barbara, and are available from 1981 to the present. This dataset is extremely useful in drought monitoring and assessing global environmental change over land. The satellite data are calibrated with ground station observations to create the final product. +CHIRPS is a high-resolution global gridded rainfall dataset that combines satellite-measured precipitation with ground station data in a consistent, long time-series dataset. The data are provided by the University of California, Santa Barbara, and are available from 1981 to the present. This dataset is extremely useful in drought monitoring and assessing global environmental change over land. The satellite data are calibrated with ground station observations to create the final product. -In this exercise, we will work with the CHIRPS dataset using the pentad. A pentad represents the grouping of five days. There are six pentads in a calendar month, with five pentads of exactly five days each and one pentad with the remaining three to six days of the month. Pentads reset at the beginning of each month, and the first day of every month is the start of a new pentad. Values at a given pixel in the CHIRPS dataset represent the total precipitation in millimeters over the pentad. +In this exercise, we will work with the CHIRPS dataset using the pentad. A pentad represents the grouping of five days. There are six pentads in a calendar month, with five pentads of exactly five days each and one pentad with the remaining three to six days of the month. Pentads reset at the beginning of each month, and the first day of every month is the start of a new pentad. Values at a given pixel in the CHIRPS dataset represent the total precipitation in millimeters over the pentad. ## Filtering an Image Collection -We will start by accessing the CHIRPS Pentad collection and filtering it to create a time series for a single year. +We will start by accessing the CHIRPS Pentad collection and filtering it to create a time series for a single year. -var chirps = ee.ImageCollection('UCSB-CHG/CHIRPS/PENTAD'); -var startDate = '2019-01-01'; -var endDate = '2020-01-01'; -var yearFiltered = chirps.filter(ee.Filter.date(startDate, endDate)); +var chirps = ee.ImageCollection('UCSB-CHG/CHIRPS/PENTAD'); +var startDate = '2019-01-01'; +var endDate = '2020-01-01'; +var yearFiltered = chirps.filter(ee.Filter.date(startDate, endDate)); print(yearFiltered, 'Date-filtered CHIRPS images'); -The CHIRPS collection contains one image for every pentad. The filtered collection above is filtered to contain one year, which equates to 72 global images. If you expand the printed collection in the Console, you will be able to see the metadata for individual images; note that their date stamps indicate that they are spaced evenly every five days (Fig. F4.2.1). +The CHIRPS collection contains one image for every pentad. The filtered collection above is filtered to contain one year, which equates to 72 global images. If you expand the printed collection in the Console, you will be able to see the metadata for individual images; note that their date stamps indicate that they are spaced evenly every five days (Fig. F4.2.1). -![](F4/image25.png) +![Fig. F4.2.1 CHIRPS time series for one year](F4/image25.png) -Fig. F4.2.1 CHIRPS time series for one year -Each image’s pixel values store the total precipitation during the pentad. Without aggregation to a period that matches other datasets, these layers are not very useful. For hydrological analysis, we typically need the total precipitation for each month or for a season. Let’s aggregate this collection so that we have 12 images—one image per month, with pixel values that represent the total precipitation for that month. +Each image’s pixel values store the total precipitation during the pentad. Without aggregation to a period that matches other datasets, these layers are not very useful. For hydrological analysis, we typically need the total precipitation for each month or for a season. Let’s aggregate this collection so that we have 12 images—one image per month, with pixel values that represent the total precipitation for that month. -::: {.callout-note} -Code Checkpoint F42a. The book’s repository contains a script that shows what your code should look like at this point. +:::{.callout-note} +Code Checkpoint F42a. The book’s repository contains a script that shows what your code should look like at this point. ::: ## Working with Dates -To aggregate the time series, we need to learn how to create and manipulate dates programmatically. This section covers some functions from the ee.Date module that will be useful. +To aggregate the time series, we need to learn how to create and manipulate dates programmatically. This section covers some functions from the ee.Date module that will be useful. -The Earth Engine API has a function called ee.Date.fromYMD that is designed to create a date object from year, month, and day values. The following code snippet shows how to define a variable containing the year value and create a date object from it. Paste the following code in a new script: +The Earth Engine API has a function called ee.Date.fromYMD that is designed to create a date object from year, month, and day values. The following code snippet shows how to define a variable containing the year value and create a date object from it. Paste the following code in a new script: -var chirps = ee.ImageCollection('UCSB-CHG/CHIRPS/PENTAD'); -var year = 2019; -var startDate = ee.Date.fromYMD(year, 1, 1); +var chirps = ee.ImageCollection('UCSB-CHG/CHIRPS/PENTAD'); +var year = 2019; +var startDate = ee.Date.fromYMD(year, 1, 1); -Now, let’s determine how to create an end date in order to be able to specify a desired time interval. The preferred way to create a date relative to another date is using the advance function. It takes two parameters—a delta value and the unit of time—and returns a new date. The code below shows how to create a date one year in the future from a given date. Paste it into your script. +Now, let’s determine how to create an end date in order to be able to specify a desired time interval. The preferred way to create a date relative to another date is using the advance function. It takes two parameters—a delta value and the unit of time—and returns a new date. The code below shows how to create a date one year in the future from a given date. Paste it into your script. -var endDate = startDate.advance(1, 'year'); +var endDate = startDate.advance(1, 'year'); -Next, paste the code below to perform filtering of the CHIRPS data using these calculated dates. After running it, check that you had accurately set the dates by looking for the dates of the images inside the printed result.. +Next, paste the code below to perform filtering of the CHIRPS data using these calculated dates. After running it, check that you had accurately set the dates by looking for the dates of the images inside the printed result.. -var yearFiltered = chirps -   .filter(ee.Filter.date(startDate, endDate)); +var yearFiltered = chirps + .filter(ee.Filter.date(startDate, endDate)); print(yearFiltered, 'Date-filtered CHIRPS images'); -Another date function that is very commonly used across Earth Engine is millis. This function takes a date object and returns the number of milliseconds since the arbitrary reference date of the start of the year 1970: 1970-01-01T00:00:00Z. This is known as the “Unix Timestamp”; it is a standard way to convert dates to numbers and allows for easy comparison between dates with high precision. Earth Engine objects store the timestamps for images and features in special properties called system:time_start and system:time_end. Both of these properties need to be supplied with a number instead of dates, and the millis function can help you do that. You can print the result of calling this function and check for yourself. +Another date function that is very commonly used across Earth Engine is millis. This function takes a date object and returns the number of milliseconds since the arbitrary reference date of the start of the year 1970: 1970-01-01T00:00:00Z. This is known as the “Unix Timestamp”; it is a standard way to convert dates to numbers and allows for easy comparison between dates with high precision. Earth Engine objects store the timestamps for images and features in special properties called system:time_start and system:time_end. Both of these properties need to be supplied with a number instead of dates, and the millis function can help you do that. You can print the result of calling this function and check for yourself. print(startDate, 'Start date'); print(endDate, 'End date'); @@ -541,93 +550,101 @@ print(endDate, 'End date'); print('Start date as timestamp', startDate.millis()); print('End date as timestamp', endDate.millis()); -We will use the millis function in the next section when we need to set the system:time_start and system:time_end properties of the aggregated images. +We will use the millis function in the next section when we need to set the system:time_start and system:time_end properties of the aggregated images. -::: {.callout-note} -Code Checkpoint F42b. The book’s repository contains a script that shows what your code should look like at this point. +:::{.callout-note} +Code Checkpoint F42b. The book’s repository contains a script that shows what your code should look like at this point. ::: ## Aggregating Images -Now we can start aggregating the pentads into monthly sums. The process of aggregation has two fundamental steps. The first is to determine the beginning and ending dates of one time interval (in this case, one month), and the second is to sum up all of the values (in this case, the pentads) that fall within each interval. To begin, we can envision that the resulting series will contain 12 images. To prepare to create an image for each month, we create an ee.List of values from 1 to 12. We can use the ee.List.sequence function, as first presented in Chap. F1.0, to create the list of items of type ee.Number. Continuing with the script of the previous section, paste the following code: +Now we can start aggregating the pentads into monthly sums. The process of aggregation has two fundamental steps. The first is to determine the beginning and ending dates of one time interval (in this case, one month), and the second is to sum up all of the values (in this case, the pentads) that fall within each interval. To begin, we can envision that the resulting series will contain 12 images. To prepare to create an image for each month, we create an ee.List of values from 1 to 12. We can use the ee.List.sequence function, as first presented in Chap. F1.0, to create the list of items of type ee.Number. Continuing with the script of the previous section, paste the following code: +```js // Aggregate this time series to compute monthly images. // Create a list of months -var months = ee.List.sequence(1, 12); +var months = ee.List.sequence(1, 12); -Next, we write a function that takes a single month as the input and returns an aggregated image for that month. Given beginningMonth as an input parameter, we first create a start and end date for that month based on the year and month variables. Then we filter the collection to find all images for that month. To create a monthly precipitation image, we apply ee.Reducer.sum to reduce the six pentad images for a month to a single image holding the summed value across the pentads. We also expressly set the timestamp properties system:time_start and system:time_end of the resulting summed image. We can also set year and month, which will help us filter the resulting collection later. +``` +Next, we write a function that takes a single month as the input and returns an aggregated image for that month. Given beginningMonth as an input parameter, we first create a start and end date for that month based on the year and month variables. Then we filter the collection to find all images for that month. To create a monthly precipitation image, we apply ee.Reducer.sum to reduce the six pentad images for a month to a single image holding the summed value across the pentads. We also expressly set the timestamp properties system:time_start and system:time_end of the resulting summed image. We can also set year and month, which will help us filter the resulting collection later. +```js // Write a function that takes a month number // and returns a monthly image. -var createMonthlyImage = function(beginningMonth) {   var startDate = ee.Date.fromYMD(year, beginningMonth, 1);   var endDate = startDate.advance(1, 'month');   var monthFiltered = yearFiltered -       .filter(ee.Filter.date(startDate, endDate));   // Calculate total precipitation.   var total = monthFiltered.reduce(ee.Reducer.sum());   return total.set({       'system:time_start': startDate.millis(),       'system:time_end': endDate.millis(),       'year': year,       'month': beginningMonth   }); +var createMonthlyImage = function(beginningMonth) { var startDate = ee.Date.fromYMD(year, beginningMonth, 1); var endDate = startDate.advance(1, 'month'); var monthFiltered = yearFiltered + .filter(ee.Filter.date(startDate, endDate)); // Calculate total precipitation. var total = monthFiltered.reduce(ee.Reducer.sum()); return total.set({ 'system:time_start': startDate.millis(), 'system:time_end': endDate.millis(), 'year': year, 'month': beginningMonth }); }; -We now have an ee.List containing items of type ee.Number from 1 to 12, with a function that can compute a monthly aggregated image for each month number. All that is left to do is to map the function over the list. As described in Chaps. F4.0 and F4.1, the map function passes over each image in the list and runs createMonthlyImage. The function first receives the number “1” and executes, returning an image to Earth Engine. Then it runs on the number “2”, and so on for all 12 numbers. The result is a list of monthly images for each month of the year. +``` +We now have an ee.List containing items of type ee.Number from 1 to 12, with a function that can compute a monthly aggregated image for each month number. All that is left to do is to map the function over the list. As described in Chaps. F4.0 and F4.1, the map function passes over each image in the list and runs createMonthlyImage. The function first receives the number “1” and executes, returning an image to Earth Engine. Then it runs on the number “2”, and so on for all 12 numbers. The result is a list of monthly images for each month of the year. +```js // map() the function on the list of months // This creates a list with images for each month in the list -var monthlyImages = months.map(createMonthlyImage); +var monthlyImages = months.map(createMonthlyImage); -We can create an ImageCollection from this ee.List of images using the ee.ImageCollection.fromImages function. +``` +We can create an ImageCollection from this ee.List of images using the ee.ImageCollection.fromImages function. +```js // Create an ee.ImageCollection. -var monthlyCollection = ee.ImageCollection.fromImages(monthlyImages); +var monthlyCollection = ee.ImageCollection.fromImages(monthlyImages); print(monthlyCollection); -We have now successfully computed an aggregated collection from the source ImageCollection by filtering, mapping, and reducing, as described in Chaps. F4.0 and F4.1. Expand the printed collection in the Console and you can verify that we now have 12 images in the newly created ImageCollection (Fig. F4.2.2). +``` +We have now successfully computed an aggregated collection from the source ImageCollection by filtering, mapping, and reducing, as described in Chaps. F4.0 and F4.1. Expand the printed collection in the Console and you can verify that we now have 12 images in the newly created ImageCollection (Fig. F4.2.2). -![](F4/image90.png) +![Fig. F4.2.2 Aggregated time series](F4/image90.png) -Fig. F4.2.2 Aggregated time series -::: {.callout-note} -Code Checkpoint F42c. The book’s repository contains a script that shows what your code should look like at this point. +:::{.callout-note} +Code Checkpoint F42c. The book’s repository contains a script that shows what your code should look like at this point. ::: ## Plotting Time Series One useful application of gridded precipitation datasets is to analyze rainfall patterns. We can plot a time-series chart for a location using the newly computed time series. We can plot the pixel value at any given point or polygon. Here we create a point geometry for a given coordinate. Continuing with the script of the previous section, paste the following code: +```js // Create a point with coordinates for the city of Bengaluru, India. -var point = ee.Geometry.Point(77.5946, 12.9716); +var point = ee.Geometry.Point(77.5946, 12.9716); -Earth Engine comes with a built-in ui.Chart.image.series function that can plot time series. In addition to the imageCollection and region parameters, we need to supply a scale value. The CHIRPS data catalog page indicates that the resolution of the data is ​​5566 meters, so we can use that as the scale. The resulting chart is printed in the Console. +``` +Earth Engine comes with a built-in ui.Chart.image.series function that can plot time series. In addition to the imageCollection and region parameters, we need to supply a scale value. The CHIRPS data catalog page indicates that the resolution of the data is ​​5566 meters, so we can use that as the scale. The resulting chart is printed in the Console. -var chart = ui.Chart.image.series({ -   imageCollection: monthlyCollection, -   region: point, -   reducer: ee.Reducer.mean(), -   scale: 5566, +var chart = ui.Chart.image.series({ + imageCollection: monthlyCollection, + region: point, + reducer: ee.Reducer.mean(), + scale: 5566, }); print(chart); -We can make the chart more informative by adding axis labels and a title. The setOptions function allows us to customize the chart using parameters from Google Charts. To customize the chart, paste the code below at the bottom of your script. The effect will be to see two charts in the editor: one with the old view of the data, and one with the customized chart. +We can make the chart more informative by adding axis labels and a title. The setOptions function allows us to customize the chart using parameters from Google Charts. To customize the chart, paste the code below at the bottom of your script. The effect will be to see two charts in the editor: one with the old view of the data, and one with the customized chart. -var chart = ui.Chart.image.series({ -   imageCollection: monthlyCollection, -   region: point, -   reducer: ee.Reducer.mean(), -   scale: 5566 +var chart = ui.Chart.image.series({ + imageCollection: monthlyCollection, + region: point, + reducer: ee.Reducer.mean(), + scale: 5566 }).setOptions({ -   lineWidth: 1, -   pointSize: 3, -   title: 'Monthly Rainfall at Bengaluru', -   vAxis: { -       title: 'Rainfall (mm)'   }, -   hAxis: { -       title: 'Month', -       gridlines: { -           count: 12       } -   } + lineWidth: 1, + pointSize: 3, + title: 'Monthly Rainfall at Bengaluru', + vAxis: { + title: 'Rainfall (mm)' }, + hAxis: { + title: 'Month', + gridlines: { + count: 12 } + } });print(chart); The customized chart (Fig. F4.2.3) shows the typical rainfall pattern in the city of Bengaluru, India. Bengaluru has a temperate climate, with pre-monsoon rains in April and May cooling down the city and a moderate monsoon season lasting from June to September. -![](F4/image32.png) +![Fig. F4.2.3 Monthly rainfall chart](F4/image32.png) -Fig. F4.2.3 Monthly rainfall chart -::: {.callout-note} -Code Checkpoint F42d. The book’s repository contains a script that shows what your code should look like at this point. +:::{.callout-note} +Code Checkpoint F42d. The book’s repository contains a script that shows what your code should look like at this point. ::: ## Synthesis {.unnumbered} @@ -635,23 +652,25 @@ Assignment 1. The CHIRPS collection contains data for 40 years. Aggregate the sa Instead of creating a list of months and writing a function to create monthly images, we will create a list of years and write a function to create yearly images. The code snippet below will help you get started. -var chirps = ee.ImageCollection('UCSB-CHG/CHIRPS/PENTAD'); +var chirps = ee.ImageCollection('UCSB-CHG/CHIRPS/PENTAD'); +```js // Create a list of years -var years = ee.List.sequence(1981, 2021); +var years = ee.List.sequence(1981, 2021); // Write a function that takes a year number // and returns a yearly image -var createYearlyImage = function(beginningYear) {   // Add your code +var createYearlyImage = function(beginningYear) { // Add your code }; -var yearlyImages = years.map(createYearlyImage); -var yearlyCollection = ee.ImageCollection.fromImages(yearlyImages); +var yearlyImages = years.map(createYearlyImage); +var yearlyCollection = ee.ImageCollection.fromImages(yearlyImages); print(yearlyCollection); +``` ## Conclusion {.unnumbered} -In this chapter, you learned how to aggregate a collection to months and plot the resulting time series for a location. This chapter also introduced useful functions for working with the dates that will be used across many different applications. You also learned how to iterate over a list using the map function. The technique of mapping a function over a list or collection is essential for processing data. Mastering this technique will allow you to scale your analysis using the parallel computing capabilities of Earth Engine. +In this chapter, you learned how to aggregate a collection to months and plot the resulting time series for a location. This chapter also introduced useful functions for working with the dates that will be used across many different applications. You also learned how to iterate over a list using the map function. The technique of mapping a function over a list or collection is essential for processing data. Mastering this technique will allow you to scale your analysis using the parallel computing capabilities of Earth Engine. ## References {.unnumbered} @@ -669,7 +688,7 @@ Okamoto K, Ushio T, Iguchi T, et al (2005) The global satellite mapping of preci -::: {.callout-tip} +:::{.callout-tip} # Chapter Information ## Author {.unlisted .unnumbered} @@ -683,7 +702,7 @@ Txomin Hermosilla, Saverio Francini, Andréa P. Nicolau, Michael A. Wulder, Joan ## Overview {.unlisted .unnumbered} -The purpose of this chapter is to provide necessary context and demonstrate different approaches for image composite generation when using data quality flags, using an initial example of removing cloud cover. We will examine different filtering options, demonstrate an approach for cloud masking, and provide additional opportunities for image composite development. Pixel selection for composite development can exclude unwanted pixels—such as those impacted by cloud, shadow, and smoke or haze—and can also preferentially select pixels based upon proximity to a target date or a preferred sensor type.   +The purpose of this chapter is to provide necessary context and demonstrate different approaches for image composite generation when using data quality flags, using an initial example of removing cloud cover. We will examine different filtering options, demonstrate an approach for cloud masking, and provide additional opportunities for image composite development. Pixel selection for composite development can exclude unwanted pixels—such as those impacted by cloud, shadow, and smoke or haze—and can also preferentially select pixels based upon proximity to a target date or a preferred sensor type. ## Learning Outcomes {.unlisted .unnumbered} @@ -700,321 +719,329 @@ The purpose of this chapter is to provide necessary context and demonstrate di * Use band scaling factors (Chap. F3.1). * Perform pixel-based transformations (Chap. F3.1). * Use neighborhood-based image transformations (Chap. F3.2). -* Write a function and map it over an ImageCollection (Chap. F4.0). +* Write a function and map it over an ImageCollection (Chap. F4.0). * Summarize an ImageCollection with reducers (Chap. F4.0, Chap. F4.1). -::: +::: ## Introduction {.unlisted .unnumbered} -In many respects, satellite remote sensing is an ideal source of data for monitoring large or remote regions. However, cloud cover is one of the most common limitations of optical sensors in providing continuous time series of data for surface mapping and monitoring. This is particularly relevant in tropical, polar, mountainous, and high-latitude areas, where clouds are often present. Many studies have addressed the extent to which cloudiness can restrict the monitoring of various regions (Zhu and Woodcock 2012, 2014; Eberhardt et al. 2016; Martins et al. 2018). +In many respects, satellite remote sensing is an ideal source of data for monitoring large or remote regions. However, cloud cover is one of the most common limitations of optical sensors in providing continuous time series of data for surface mapping and monitoring. This is particularly relevant in tropical, polar, mountainous, and high-latitude areas, where clouds are often present. Many studies have addressed the extent to which cloudiness can restrict the monitoring of various regions (Zhu and Woodcock 2012, 2014; Eberhardt et al. 2016; Martins et al. 2018). -Clouds and cloud shadows reduce the view of optical sensors and completely block or obscure the spectral response from Earth’s surface (Cao et al. 2020). Working with pixels that are cloud-contaminated can significantly influence the accuracy and information content of products derived from a variety of remote sensing activities, including land cover classification, vegetation modeling, and especially change detection, where unscreened clouds might be mapped as false changes (Braaten et al. 2015, Zhu et al. 2015). Thus, the information provided by cloud detection algorithms is critical to exclude clouds and cloud shadows from subsequent processing steps. +Clouds and cloud shadows reduce the view of optical sensors and completely block or obscure the spectral response from Earth’s surface (Cao et al. 2020). Working with pixels that are cloud-contaminated can significantly influence the accuracy and information content of products derived from a variety of remote sensing activities, including land cover classification, vegetation modeling, and especially change detection, where unscreened clouds might be mapped as false changes (Braaten et al. 2015, Zhu et al. 2015). Thus, the information provided by cloud detection algorithms is critical to exclude clouds and cloud shadows from subsequent processing steps. Historically, cloud detection algorithms derived the cloud information by considering a single date-image and sun illumination geometry (Irish et al. 2006, Huang et al. 2010). In contrast, current, more accurate cloud detection algorithms are based on the analysis of Landsat time series (Zhu and Woodcock 2014, Zhu and Helmer 2018). Cloud detection algorithms inform on the presence of clouds, cloud shadows, and other atmospheric conditions (e.g., presence of snow). The presence and extent of cloud contamination within a pixel is currently provided with Landsat and Sentinel-2 imagery as ancillary data via quality flags at the pixel level. Additionally, quality flags also inform on other acquisition-related conditions, including radiometric saturation and terrain occlusion, which enables us to assess the usefulness and convenience of inclusion of each pixel in subsequent analyses. The quality flags are ideally suited to reduce users’ manual supervision and maximize the automatic processing approaches. Most automated algorithms (for classification or change detection, for example) work best on images free of clouds and cloud shadows, that cover the full area without spatial or spectral inconsistencies. Thus, the image representation over the study area should be seamless, containing as few data gaps as possible. Image compositing techniques are primarily used to reduce the impact of clouds and cloud shadows, as well as aerosol contamination, view angle effects, and data volumes (White et al. 2014). Compositing approaches typically rely on the outputs of cloud detection algorithms and quality flags to include or exclude pixels from the resulting composite products (Roy et al. 2010). Epochal image composites help overcome the limited availability of cloud-free imagery in some areas, and are constructed by considering the pixels from all images acquired in a given period (e.g., season, year). -The information provided by the cloud masks and pixel flags guides the establishment of rules to rank the quality of the pixels based on the presence of and distance to clouds, cloud shadows, or atmospheric haze (Griffiths et al. 2010). Higher scores are assigned to pixels with more desirable conditions, based on the presence of clouds and also other acquisition circumstances, such as acquisition date or sensor. Those pixels with the highest scores are included in the subsequent composite development. Image compositing approaches enable users to define the rules that are most appropriate for their particular information needs and study area to generate imagery covering large areas instead of being limited to the analysis of single scenes (Hermosilla et al. 2015, Loveland and Dwyer 2012). Moreover, generating image composites at regular intervals (e.g., annually) allows for the analysis of long temporal series over large areas, fulfilling a critical information need for monitoring programs. +The information provided by the cloud masks and pixel flags guides the establishment of rules to rank the quality of the pixels based on the presence of and distance to clouds, cloud shadows, or atmospheric haze (Griffiths et al. 2010). Higher scores are assigned to pixels with more desirable conditions, based on the presence of clouds and also other acquisition circumstances, such as acquisition date or sensor. Those pixels with the highest scores are included in the subsequent composite development. Image compositing approaches enable users to define the rules that are most appropriate for their particular information needs and study area to generate imagery covering large areas instead of being limited to the analysis of single scenes (Hermosilla et al. 2015, Loveland and Dwyer 2012). Moreover, generating image composites at regular intervals (e.g., annually) allows for the analysis of long temporal series over large areas, fulfilling a critical information need for monitoring programs. The general workflow to generate a cloud-free composite involves: -1. Defining your area of interest (AOI). -2. Filtering (ee.Filter) the satellite ImageCollection to desired parameters. +1. Defining your area of interest (AOI). +2. Filtering (ee.Filter) the satellite ImageCollection to desired parameters. 3. Applying a cloud mask. 4. Reducing (ee.Reducer) the collection to generate a composite. -5. Using the GEE-BAP application to generate annual best-available-pixel image composites by globally combining multiple Landsat sensors and images. +5. Using the GEE-BAP application to generate annual best-available-pixel image composites by globally combining multiple Landsat sensors and images. Additional steps may be necessary to improve the composite generated. These steps will be explained in the following sections. -## Cloud Filter and Cloud Mask +## Cloud Filter and Cloud Mask -The first step is to define your AOI and center the map. The goal is to create a nationwide composite for the country of Colombia. We will use the Large Scale International Boundary (2017) simplified dataset from the US Department of State (USDOS), which contains polygons for all countries of the world. +The first step is to define your AOI and center the map. The goal is to create a nationwide composite for the country of Colombia. We will use the Large Scale International Boundary (2017) simplified dataset from the US Department of State (USDOS), which contains polygons for all countries of the world. +```js // ---------- Section 1 ----------------- // Define the AOI. -var country = ee.FeatureCollection('USDOS/LSIB_SIMPLE/2017') -   .filter(ee.Filter.equals('country_na', 'Colombia')); +var country = ee.FeatureCollection('USDOS/LSIB_SIMPLE/2017') + .filter(ee.Filter.equals('country_na', 'Colombia')); // Center the Map. The second parameter is zoom level. Map.centerObject(country, 5); -We will start creating a composite from the Landsat 8 collection. First, we define two time variables: startDate and endDate. Here, we will create a composite for the year 2019. Then, we will define a collection for the Landsat 8 Level 2, Collection 2, Tier 1 variable and filter it to our AOI and time period. We define and use a function to apply scaling factors to the Landsat 8 Collection 2 data. +``` +We will start creating a composite from the Landsat 8 collection. First, we define two time variables: startDate and endDate. Here, we will create a composite for the year 2019. Then, we will define a collection for the Landsat 8 Level 2, Collection 2, Tier 1 variable and filter it to our AOI and time period. We define and use a function to apply scaling factors to the Landsat 8 Collection 2 data. +```js // Define time variables. -var startDate = '2019-01-01'; -var endDate = '2019-12-31'; +var startDate = '2019-01-01'; +var endDate = '2019-12-31'; // Load and filter the Landsat 8 collection. -var landsat8 = ee.ImageCollection('LANDSAT/LC08/C02/T1_L2') -   .filterBounds(country) -   .filterDate(startDate, endDate); +var landsat8 = ee.ImageCollection('LANDSAT/LC08/C02/T1_L2') + .filterBounds(country) + .filterDate(startDate, endDate); // Apply scaling factors. -function applyScaleFactors(image) {   var opticalBands = image.select('SR_B.').multiply(0.0000275).add(-       0.2);   var thermalBands = image.select('ST_B.*').multiply(0.00341802) -       .add(149.0);   return image.addBands(opticalBands, null, true) -       .addBands(thermalBands, null, true); +function applyScaleFactors(image) { var opticalBands = image.select('SR_B.').multiply(0.0000275).add(- 0.2); var thermalBands = image.select('ST_B.*').multiply(0.00341802) + .add(149.0); return image.addBands(opticalBands, null, true) + .addBands(thermalBands, null, true); } landsat8 = landsat8.map(applyScaleFactors); -Now, we can create a composite. We will use the median function, which has the same effect as writing reduce(ee.Reducer.median()) as seen in Chap. F4.0, to reduce our ImageCollection to a median composite. Add the resulting composite to the map using visualization parameters. +``` +Now, we can create a composite. We will use the median function, which has the same effect as writing reduce(ee.Reducer.median()) as seen in Chap. F4.0, to reduce our ImageCollection to a median composite. Add the resulting composite to the map using visualization parameters. +```js // Create composite. -var composite = landsat8.median().clip(country); +var composite = landsat8.median().clip(country); -var visParams = { -   bands: ['SR_B4', 'SR_B3', 'SR_B2'], -   min: 0, -   max: 0.2 +var visParams = { + bands: ['SR_B4', 'SR_B3', 'SR_B2'], + min: 0, + max: 0.2 }; Map.addLayer(composite, visParams, 'L8 Composite'); -![](F4/image12.png) +``` +![Fig. F4.3.1 Landsat 8 surface reflectance 2019 median composite of Colombia](F4/image12.png) -Fig. F4.3.1 Landsat 8 surface reflectance 2019 median composite of Colombia -The resulting composite (Fig. F4.3.1) has lots of clouds, especially in the western, mountainous regions of Colombia. In tropical regions, it is very challenging to generate a high-quality, cloud-free composite without first filtering images for cloud cover, even if our collection is constrained to only include images acquired during the dry season. Therefore, let’s filter our collection by the CLOUD_COVER parameter to avoid cloudy images. We will start with images that have less than 50% cloud cover. +The resulting composite (Fig. F4.3.1) has lots of clouds, especially in the western, mountainous regions of Colombia. In tropical regions, it is very challenging to generate a high-quality, cloud-free composite without first filtering images for cloud cover, even if our collection is constrained to only include images acquired during the dry season. Therefore, let’s filter our collection by the CLOUD_COVER parameter to avoid cloudy images. We will start with images that have less than 50% cloud cover. +```js // Filter by the CLOUD_COVER property. -var landsat8FiltClouds = landsat8 -   .filterBounds(country) -   .filterDate(startDate, endDate) -   .filter(ee.Filter.lessThan('CLOUD_COVER', 50)); +var landsat8FiltClouds = landsat8 + .filterBounds(country) + .filterDate(startDate, endDate) + .filter(ee.Filter.lessThan('CLOUD_COVER', 50)); // Create a composite from the filtered imagery. -var compositeFiltClouds = landsat8FiltClouds.median().clip(country); +var compositeFiltClouds = landsat8FiltClouds.median().clip(country); -Map.addLayer(compositeFiltClouds, visParams,   'L8 Composite cloud filter'); +Map.addLayer(compositeFiltClouds, visParams, 'L8 Composite cloud filter'); // Print size of collections, for comparison. print('Size landsat8 collection', landsat8.size()); print('Size landsat8FiltClouds collection', landsat8FiltClouds.size()); -![](F4/image50.png) +``` +![Fig. F4.3.2 Landsat 8 surface reflectance 2019 median composite of Colombia filtered by cloud cover less than 50%](F4/image50.png) -Fig. F4.3.2 Landsat 8 surface reflectance 2019 median composite of Colombia filtered by cloud cover less than 50% -This new composite (Fig. F4.3.2) looks slightly better than the previous one, but still very cloudy. Remember to turn off the first layer or adjust the transparency to visualize only this new composite. The code prints the size of these collections, using the size function) to see how many images were left out after we applied the cloud cover threshold. (There are 1201 images in the landsat8 collection, compared to 493 in the landsat8FiltClouds collection—a lot of scenes with cloud cover greater than or equal to 50%.) +This new composite (Fig. F4.3.2) looks slightly better than the previous one, but still very cloudy. Remember to turn off the first layer or adjust the transparency to visualize only this new composite. The code prints the size of these collections, using the size function) to see how many images were left out after we applied the cloud cover threshold. (There are 1201 images in the landsat8 collection, compared to 493 in the landsat8FiltClouds collection—a lot of scenes with cloud cover greater than or equal to 50%.) -Try adjusting the CLOUD_COVER threshold in the landsat8FiltClouds variable to different percentages and checking the results. For example, with 20% set as the threshold (Fig. F4.3.3), you can see that many parts of the country have image gaps. (Remember to turn off the first layer or adjust its transparency; you can also set the shown parameter in the Map.addLayer function to false so the layer does not automatically load). So there is a trade-off between a stricter cloud cover threshold and data availability. Additionally, even with a cloud filter, some tiles still present a large area cover of clouds. +Try adjusting the CLOUD_COVER threshold in the landsat8FiltClouds variable to different percentages and checking the results. For example, with 20% set as the threshold (Fig. F4.3.3), you can see that many parts of the country have image gaps. (Remember to turn off the first layer or adjust its transparency; you can also set the shown parameter in the Map.addLayer function to false so the layer does not automatically load). So there is a trade-off between a stricter cloud cover threshold and data availability. Additionally, even with a cloud filter, some tiles still present a large area cover of clouds. -![](F4/image42.png) +![Fig. F4.3.3 Landsat 8 surface reflectance 2019 median composite of Colombia filtered by cloud cover less than 20%](F4/image42.png) -Fig. F4.3.3 Landsat 8 surface reflectance 2019 median composite of Colombia filtered by cloud cover less than 20% -This is due to persistent cloud cover in some regions of Colombia. However, a cloud mask can be applied to improve the results. The Landsat 8 Collection 2 contains a quality assessment (QA) band called QA_PIXEL that provides useful information on certain conditions within the data, and allows users to apply per-pixel filters. Each pixel in the QA band contains unsigned integers that represent bit-packed combinations of surface, atmospheric, and sensor conditions. +This is due to persistent cloud cover in some regions of Colombia. However, a cloud mask can be applied to improve the results. The Landsat 8 Collection 2 contains a quality assessment (QA) band called QA_PIXEL that provides useful information on certain conditions within the data, and allows users to apply per-pixel filters. Each pixel in the QA band contains unsigned integers that represent bit-packed combinations of surface, atmospheric, and sensor conditions. -We will also make use of the QA_RADSAT band, which indicates which bands are radiometrically saturated. A pixel value of 1 means saturated, so we will be masking these pixels. +We will also make use of the QA_RADSAT band, which indicates which bands are radiometrically saturated. A pixel value of 1 means saturated, so we will be masking these pixels. -As described in Chap. F4.0, we will create a function to apply a cloud mask to an image, and then map this function over our collection. The mask is applied by using the updateMask function. This function “eliminates” undesired pixels from the analysis, i.e., makes them transparent, by taking the mask as the input. You will see that this cloud mask function (or similar versions) is used in other chapters of the book. Note: Remember to set the cloud cover threshold back to 50 in the landsat8FiltClouds variable. +As described in Chap. F4.0, we will create a function to apply a cloud mask to an image, and then map this function over our collection. The mask is applied by using the updateMask function. This function “eliminates” undesired pixels from the analysis, i.e., makes them transparent, by taking the mask as the input. You will see that this cloud mask function (or similar versions) is used in other chapters of the book. Note: Remember to set the cloud cover threshold back to 50 in the landsat8FiltClouds variable. +```js // Define the cloud mask function. -function maskSrClouds(image) {   // Bit 0 - Fill   // Bit 1 - Dilated Cloud   // Bit 2 - Cirrus   // Bit 3 - Cloud   // Bit 4 - Cloud Shadow   var qaMask = image.select('QA_PIXEL').bitwiseAnd(parseInt('11111',       2)).eq(0);   var saturationMask = image.select('QA_RADSAT').eq(0);   return image.updateMask(qaMask) -       .updateMask(saturationMask); +function maskSrClouds(image) { // Bit 0 - Fill // Bit 1 - Dilated Cloud // Bit 2 - Cirrus // Bit 3 - Cloud // Bit 4 - Cloud Shadow var qaMask = image.select('QA_PIXEL').bitwiseAnd(parseInt('11111', 2)).eq(0); var saturationMask = image.select('QA_RADSAT').eq(0); return image.updateMask(qaMask) + .updateMask(saturationMask); } // Apply the cloud mask to the collection. -var landsat8FiltMasked = landsat8FiltClouds.map(maskSrClouds); +var landsat8FiltMasked = landsat8FiltClouds.map(maskSrClouds); // Create a composite. -var landsat8compositeMasked = landsat8FiltMasked.median().clip(country); +var landsat8compositeMasked = landsat8FiltMasked.median().clip(country); Map.addLayer(landsat8compositeMasked, visParams, 'L8 composite masked'); -![](F4/image39.png) +``` +![Fig. F4.3.4 Landsat 8 surface reflectance 2019 median composite of Colombia filtered by cloud cover less than 50% and with cloud mask applied](F4/image39.png) -Fig. F4.3.4 Landsat 8 surface reflectance 2019 median composite of Colombia filtered by cloud cover less than 50% and with cloud mask applied -Because we are dealing with bits, in the maskSrClouds function we utilized the bitwiseAnd and parseInt functions. These are functions that serve the purpose of unpacking the bit information. A bitwise AND is a binary operation that takes two equal-length binary representations and performs the logical AND operation on each pair of corresponding bits. Thus, if both bits in the compared positions have the value 1, the bit in the resulting binary representation is 1 (1 × 1 = 1); otherwise, the result is 0 (1 × 0 = 0 and 0 × 0 = 0). The parseInt function parses a string argument (in our case, five-character string '11111') and returns an integer of the specified numbering system, base 2. +Because we are dealing with bits, in the maskSrClouds function we utilized the bitwiseAnd and parseInt functions. These are functions that serve the purpose of unpacking the bit information. A bitwise AND is a binary operation that takes two equal-length binary representations and performs the logical AND operation on each pair of corresponding bits. Thus, if both bits in the compared positions have the value 1, the bit in the resulting binary representation is 1 (1 × 1 = 1); otherwise, the result is 0 (1 × 0 = 0 and 0 × 0 = 0). The parseInt function parses a string argument (in our case, five-character string '11111') and returns an integer of the specified numbering system, base 2. -The resulting composite (Fig. F4.3.4) shows masked clouds, and is more spatially exhaustive in coverage compared to previous composites (don’t forget to uncheck the previous layers). This is because, when compositing all the images into one, we are not taking cloudy pixels into account anymore; therefore, the resulting pixel is not cloud covered but an actual representation of the landscape. However, data gaps are still an issue due to cloud cover. If you do not specifically need an annual composite, a first approach is to create a two-year composite to try to mitigate the missing data issue, or to have a series of rules that allows for selecting pixels for that particular year (as in Sect. 3 below). Change the startDate variable to 2018-01-01 to include all images from 2018 and 2019 in the collection. How does the cloud-masked composite (Fig. F4.3.5) compare to the 2019 one? +The resulting composite (Fig. F4.3.4) shows masked clouds, and is more spatially exhaustive in coverage compared to previous composites (don’t forget to uncheck the previous layers). This is because, when compositing all the images into one, we are not taking cloudy pixels into account anymore; therefore, the resulting pixel is not cloud covered but an actual representation of the landscape. However, data gaps are still an issue due to cloud cover. If you do not specifically need an annual composite, a first approach is to create a two-year composite to try to mitigate the missing data issue, or to have a series of rules that allows for selecting pixels for that particular year (as in Sect. 3 below). Change the startDate variable to 2018-01-01 to include all images from 2018 and 2019 in the collection. How does the cloud-masked composite (Fig. F4.3.5) compare to the 2019 one? -![](F4/image85.png) +![Fig. F4.3.5 One-year, startDate variable set to 2019-01-01, (left) and two-year, startDate variable set to 2018-01-01, (right) median composites with 50% cloud cover threshold and cloud mask applied](F4/image85.png) -Fig. F4.3.5 One-year, startDate variable set to 2019-01-01, (left) and two-year, startDate variable set to 2018-01-01, (right) median composites with 50% cloud cover threshold and cloud mask applied The resulting image has substantially fewer data gaps (you can zoom in to better see them). Again, if the time period is not a constraint for the creation of your composite, you can incorporate more images from a third year, and so on. -::: {.callout-note} -Code Checkpoint F43a. The book’s repository contains a script that shows what your code should look like at this point. +:::{.callout-note} +Code Checkpoint F43a. The book’s repository contains a script that shows what your code should look like at this point. ::: ## Incorporating Data from Other Satellites -Another option to reduce the presence of data gaps in cloudy situations is to bring in imagery from other sensors acquired during the time period of interest. The Landsat collection spans multiple missions, which have continuously acquired uninterrupted data since 1972 at different acquisition dates. Next, we will try incorporating Landsat 7 Level 2, Collection 2, Tier 1 images from 2019 to fill the gaps in the 2019 Landsat 8 composite. +Another option to reduce the presence of data gaps in cloudy situations is to bring in imagery from other sensors acquired during the time period of interest. The Landsat collection spans multiple missions, which have continuously acquired uninterrupted data since 1972 at different acquisition dates. Next, we will try incorporating Landsat 7 Level 2, Collection 2, Tier 1 images from 2019 to fill the gaps in the 2019 Landsat 8 composite. -To generate a Landsat 7 composite, we apply similar steps to the ones we did for Landsat 8, so keep adding code to the same script from Sect. 1. First, define your Landsat 7 collection variable and the scaling function. Then, filter the collection, apply the cloud mask (since we know Colombia has persistent cloud cover), and apply the scaling function. Note that we will use the same cloud mask function defined above, since the bits information for Landsat 7 is the same as for Landsat 8. Finally, create the median composite. After pasting in the code below but before executing it, change the startDate variable back to 2019-01-01 in order to create a one-year composite of 2019. +To generate a Landsat 7 composite, we apply similar steps to the ones we did for Landsat 8, so keep adding code to the same script from Sect. 1. First, define your Landsat 7 collection variable and the scaling function. Then, filter the collection, apply the cloud mask (since we know Colombia has persistent cloud cover), and apply the scaling function. Note that we will use the same cloud mask function defined above, since the bits information for Landsat 7 is the same as for Landsat 8. Finally, create the median composite. After pasting in the code below but before executing it, change the startDate variable back to 2019-01-01 in order to create a one-year composite of 2019. +```js // ---------- Section 2 ----------------- // Define Landsat 7 Level 2, Collection 2, Tier 1 collection. -var landsat7 = ee.ImageCollection('LANDSAT/LE07/C02/T1_L2'); +var landsat7 = ee.ImageCollection('LANDSAT/LE07/C02/T1_L2'); // Scaling factors for L7. -function applyScaleFactorsL7(image) {   var opticalBands = image.select('SR_B.').multiply(0.0000275).add(-       0.2);   var thermalBand = image.select('ST_B6').multiply(0.00341802).add(       149.0);   return image.addBands(opticalBands, null, true) -       .addBands(thermalBand, null, true); +function applyScaleFactorsL7(image) { var opticalBands = image.select('SR_B.').multiply(0.0000275).add(- 0.2); var thermalBand = image.select('ST_B6').multiply(0.00341802).add( 149.0); return image.addBands(opticalBands, null, true) + .addBands(thermalBand, null, true); } // Filter collection, apply cloud mask, and scaling factors. -var landsat7FiltMasked = landsat7 -   .filterBounds(country) -   .filterDate(startDate, endDate) -   .filter(ee.Filter.lessThan('CLOUD_COVER', 50)) -   .map(maskSrClouds) -   .map(applyScaleFactorsL7); +var landsat7FiltMasked = landsat7 + .filterBounds(country) + .filterDate(startDate, endDate) + .filter(ee.Filter.lessThan('CLOUD_COVER', 50)) + .map(maskSrClouds) + .map(applyScaleFactorsL7); // Create composite. -var landsat7compositeMasked = landsat7FiltMasked -   .median() -   .clip(country); +var landsat7compositeMasked = landsat7FiltMasked + .median() + .clip(country); Map.addLayer(landsat7compositeMasked, -   { -       bands: ['SR_B3', 'SR_B2', 'SR_B1'], -       min: 0, -       max: 0.2   },   'L7 composite masked'); + { + bands: ['SR_B3', 'SR_B2', 'SR_B1'], + min: 0, + max: 0.2 }, 'L7 composite masked'); -![](F4/image73.png) +``` +![Fig. F4.3.6 One-year Landsat 7 median composite with 50% cloud cover threshold and cloud mask applied](F4/image73.png) -Fig. F4.3.6 One-year Landsat 7 median composite with 50% cloud cover threshold and cloud mask applied -Note that we used bands: ['SR_B3', 'SR_B2', 'SR_B1'] to visualize the composite because Landsat 7 has different band designations. The sensors aboard each of the Landsat satellites were designed to acquire data in different ranges of frequencies along the electromagnetic spectrum. Whereas for Landsat 8, the red, green, and blue bands are B4, B3, and B2, respectively, for Landsat 7, these same bands are B3, B2, and B1, respectively. +Note that we used bands: ['SR_B3', 'SR_B2', 'SR_B1'] to visualize the composite because Landsat 7 has different band designations. The sensors aboard each of the Landsat satellites were designed to acquire data in different ranges of frequencies along the electromagnetic spectrum. Whereas for Landsat 8, the red, green, and blue bands are B4, B3, and B2, respectively, for Landsat 7, these same bands are B3, B2, and B1, respectively. -You should see an image with systematic gaps like the one shown in Fig. F4.3.6 (remember to turn off the other layers, and zoom in to better see the data gaps). Landsat 7 was launched in 1999, but since 2003, the sensor has acquired and delivered data with data gaps caused by a scan line corrector (SLC) failure. Without an operating SLC, the sensor’s line of sight traces a zig-zag pattern along the satellite ground track, and, as a result, the imaged area is duplicated and some areas are missed. When the Level 1 data are processed, the duplicated areas are removed, leaving data gaps (Fig. F4.3.7). For more information about Landsat 7 and SLC error, please refer to the USGS Landsat 7 page. However, even with the SLC error, we can still use the Landsat 7 data in our composite. Now, let’s combine the Landsat 7 and 8 collections. +You should see an image with systematic gaps like the one shown in Fig. F4.3.6 (remember to turn off the other layers, and zoom in to better see the data gaps). Landsat 7 was launched in 1999, but since 2003, the sensor has acquired and delivered data with data gaps caused by a scan line corrector (SLC) failure. Without an operating SLC, the sensor’s line of sight traces a zig-zag pattern along the satellite ground track, and, as a result, the imaged area is duplicated and some areas are missed. When the Level 1 data are processed, the duplicated areas are removed, leaving data gaps (Fig. F4.3.7). For more information about Landsat 7 and SLC error, please refer to the USGS Landsat 7 page. However, even with the SLC error, we can still use the Landsat 7 data in our composite. Now, let’s combine the Landsat 7 and 8 collections. -![](F4/image54.png) +![Fig. F4.3.7 Landsat 7’s SLC-off condition. Source: USGS](F4/image54.png) -Fig. F4.3.7 Landsat 7’s SLC-off condition. Source: USGS -Since Landsat 7 and 8 have different band designations, first we create a function to rename the bands from Landsat 7 to match the names used for Landsat 8 and map that function over our Landsat 7 collection. +Since Landsat 7 and 8 have different band designations, first we create a function to rename the bands from Landsat 7 to match the names used for Landsat 8 and map that function over our Landsat 7 collection. +```js // Since Landsat 7 and 8 have different band designations, // let's create a function to rename L7 bands to match to L8. -function rename(image) {   return image.select( -       ['SR_B1', 'SR_B2', 'SR_B3', 'SR_B4', 'SR_B5', 'SR_B7'], -       ['SR_B2', 'SR_B3', 'SR_B4', 'SR_B5', 'SR_B6', 'SR_B7']); +function rename(image) { return image.select( + ['SR_B1', 'SR_B2', 'SR_B3', 'SR_B4', 'SR_B5', 'SR_B7'], + ['SR_B2', 'SR_B3', 'SR_B4', 'SR_B5', 'SR_B6', 'SR_B7']); } // Apply the rename function. -var landsat7FiltMaskedRenamed = landsat7FiltMasked.map(rename); +var landsat7FiltMaskedRenamed = landsat7FiltMasked.map(rename); -If you print the first images of both the landsat7FiltMasked and landsat7FiltMaskedRenamed collections (Fig. F4.3.8), you will see that the bands got renamed, and not all bands got copied over (SR_ATMOS_OPACITY, SR_CLOUD_QA, SR_B6, etc.). To copy these additional bands, simply add them to the rename function. You will need to rename SR_B6 so it does not have the same name as the new band 5. +``` +If you print the first images of both the landsat7FiltMasked and landsat7FiltMaskedRenamed collections (Fig. F4.3.8), you will see that the bands got renamed, and not all bands got copied over (SR_ATMOS_OPACITY, SR_CLOUD_QA, SR_B6, etc.). To copy these additional bands, simply add them to the rename function. You will need to rename SR_B6 so it does not have the same name as the new band 5. -![](F4/image15.png) +![Fig. F4.3.8 First images of landsat7FiltMasked and landsat7FiltMaskedRenamed, respectively](F4/image15.png) -Fig. F4.3.8 First images of landsat7FiltMasked and landsat7FiltMaskedRenamed, respectively -Now we merge the two collections using the merge function for ImageCollection and mapping over a function to cast the Landsat 7 input values to a 32-bit float using the toFloat function for consistency. To merge collections, the number and names of the bands must be the same in each collection. We use the select function (Chap. F1.1) to select the Landsat 8 bands to be the same as Landsat 7’s. When creating the new Landsat 7 and 8 composite, if we did not select these 6 bands, we would get an error message for trying to composite a collection that has 6 bands (Landsat 7) with a collection that has 19 bands (Landsat 8). +Now we merge the two collections using the merge function for ImageCollection and mapping over a function to cast the Landsat 7 input values to a 32-bit float using the toFloat function for consistency. To merge collections, the number and names of the bands must be the same in each collection. We use the select function (Chap. F1.1) to select the Landsat 8 bands to be the same as Landsat 7’s. When creating the new Landsat 7 and 8 composite, if we did not select these 6 bands, we would get an error message for trying to composite a collection that has 6 bands (Landsat 7) with a collection that has 19 bands (Landsat 8). +```js // Merge Landsat collections. -var landsat78 = landsat7FiltMaskedRenamed -   .merge(landsat8FiltMasked.select( -       ['SR_B2', 'SR_B3', 'SR_B4', 'SR_B5', 'SR_B6', 'SR_B7'])) -   .map(function(img) {       return img.toFloat(); -   }); +var landsat78 = landsat7FiltMaskedRenamed + .merge(landsat8FiltMasked.select( + ['SR_B2', 'SR_B3', 'SR_B4', 'SR_B5', 'SR_B6', 'SR_B7'])) + .map(function(img) { return img.toFloat(); + }); print('Merged collections', landsat78); -Now we have a collection with about 1000 images. Next, we will take the median of the values across the ImageCollection.  +``` +Now we have a collection with about 1000 images. Next, we will take the median of the values across the ImageCollection. +```js // Create Landsat 7 and 8 image composite and add to the Map. -var landsat78composite = landsat78.median().clip(country); +var landsat78composite = landsat78.median().clip(country); Map.addLayer(landsat78composite, visParams, 'L7 and L8 composite'); +``` Comparing the composite generated considering both Landsat 7 and 8 to the Landsat 8-only composite, it is evident that there is a reduction in the amount of data gaps in the final result (Fig. F4.3.9). The resulting Landsat 7 and 8 image composite still has data gaps due to the presence of clouds and Landsat 7’s SLC-off data. You can try setting the center of the map to the point with latitude 3.6023 and longitude −75.0741 to see the inset example of Fig. F4.3.9. -![](F4/image30.png) +![Fig. F4.3.9 Landsat 8-only composite (left) and Landsat 7 and 8 composite (right) for 2019. Inset centered at latitude 3.6023, longitude −75.0741.](F4/image30.png) -Fig. F4.3.9 Landsat 8-only composite (left) and Landsat 7 and 8 composite (right) for 2019. Inset centered at latitude 3.6023, longitude −75.0741. -::: {.callout-note} -Code Checkpoint F43b. The book’s repository contains a script that shows what your code should look like at this point. +:::{.callout-note} +Code Checkpoint F43b. The book’s repository contains a script that shows what your code should look like at this point. ::: ## Best-Available-Pixel Compositing Earth Engine Application This section presents an Earth Engine application that enables the generation of annual best-available-pixel (BAP) image composites by globally combining multiple Landsat sensors and images: GEE-BAP. Annual BAP image composites are generated by choosing optimal observations for each pixel from all available Landsat 5 TM, Landsat 7 ETM+, and Landsat 8 OLI imagery within a given year and within a given day range from a specified acquisition day of year, in addition to other constraints defined by the user. The data accessible via Earth Engine are from the USGS free and open archive of Landsat data. The Landsat images used are atmospherically corrected to surface reflectance values. Following White et al. (2014), a series of scoring functions ranks each pixel observation for (1) acquisition day of year, (2) cloud cover in the scene, (3) distance to clouds and cloud shadows, (4) presence of haze, and (5) acquisition sensor. Further information on the BAP image compositing approach can be found in Griffiths et al. (2013), and detailed information on tuning parameters can be found in White et al. (2014). -::: {.callout-note} -Code Checkpoint F43c. The book’s repository contains information about accessing the GEE-BAP interface and its related functions. +:::{.callout-note} +Code Checkpoint F43c. The book’s repository contains information about accessing the GEE-BAP interface and its related functions. ::: -![](F4/image70.png)![](F4/image81.png)![](F4/image83.png) +![Fig. F4.3.10 GEE-BAP user interface controls](F4/image70.png)![Fig. F4.3.10 GEE-BAP user interface controls](F4/image81.png)![Fig. F4.3.10 GEE-BAP user interface controls](F4/image83.png) -Fig. F4.3.10 GEE-BAP user interface controls -Once you have loaded the GEE-BAP interface (Fig. F4.3.10) using the instructions in the Code Checkpoint, you will notice that it is divided into three sections: (1) Input/Output options, (2) Pixel scoring options, and (3) Advanced parameters. Users indicate the study area, the time period for generating annual BAP composites (i.e., start and end years), and the path to store the results in the Input/Output options. Users have three options to define the study area. The Draw study area option uses the Draw a shape and Draw a rectangle tools to define the area of interest. The Upload image template option utilizes an image template uploaded by the user in TIFF format. This option is well suited to generating BAP composites that match the projection, pixel size, and extent to existing raster datasets. The Work globally option generates BAP composites for the entire globe; note that when this option is selected, complete data download is not available due to the Earth’s size. With Start year and End year, users can indicate the beginning and end of the annual time series of BAP image composites to be generated. Multiple image composites are then generated—one composite for each year—resulting in a time series of annual composites. For each year, composites are uniquely generated utilizing images acquired on the days within the specified Date range. Produced BAP composites can be saved in the indicated (Path) Google Drive folder using the Tasks tab. Results are generated in a tiled, TIFF format, accompanied by a CSV file that indicates the parameters used to construct the composite. +Once you have loaded the GEE-BAP interface (Fig. F4.3.10) using the instructions in the Code Checkpoint, you will notice that it is divided into three sections: (1) Input/Output options, (2) Pixel scoring options, and (3) Advanced parameters. Users indicate the study area, the time period for generating annual BAP composites (i.e., start and end years), and the path to store the results in the Input/Output options. Users have three options to define the study area. The Draw study area option uses the Draw a shape and Draw a rectangle tools to define the area of interest. The Upload image template option utilizes an image template uploaded by the user in TIFF format. This option is well suited to generating BAP composites that match the projection, pixel size, and extent to existing raster datasets. The Work globally option generates BAP composites for the entire globe; note that when this option is selected, complete data download is not available due to the Earth’s size. With Start year and End year, users can indicate the beginning and end of the annual time series of BAP image composites to be generated. Multiple image composites are then generated—one composite for each year—resulting in a time series of annual composites. For each year, composites are uniquely generated utilizing images acquired on the days within the specified Date range. Produced BAP composites can be saved in the indicated (Path) Google Drive folder using the Tasks tab. Results are generated in a tiled, TIFF format, accompanied by a CSV file that indicates the parameters used to construct the composite. -As noted, GEE-BAP implements five pixel scoring functions: (1) target acquisition day of year and day range, (2) maximum cloud coverage per scene, (3) distance to clouds and cloud shadows, (4) atmospheric opacity, and (5) a penalty for images acquired under the Landsat 7 ETM+ SLC-off malfunction. By defining the Acquisition day of year and Day range, those candidate pixels acquired closer to a defined acquisition day of year are ranked higher. Note that pixels acquired outside the day range window are excluded from subsequent composite development. For example, if the target day of year is defined as “08-01” and the day range as “31,” only those pixels acquired between July 1 and August 31 are considered, and the ones acquired closer to August 1 will receive a higher score. +As noted, GEE-BAP implements five pixel scoring functions: (1) target acquisition day of year and day range, (2) maximum cloud coverage per scene, (3) distance to clouds and cloud shadows, (4) atmospheric opacity, and (5) a penalty for images acquired under the Landsat 7 ETM+ SLC-off malfunction. By defining the Acquisition day of year and Day range, those candidate pixels acquired closer to a defined acquisition day of year are ranked higher. Note that pixels acquired outside the day range window are excluded from subsequent composite development. For example, if the target day of year is defined as “08-01” and the day range as “31,” only those pixels acquired between July 1 and August 31 are considered, and the ones acquired closer to August 1 will receive a higher score. -The scoring function Max cloud cover in scene indicates the maximum percentage of cloud cover in an image that will be accepted by the user in the BAP image compositing process. Defining a value of 70% implies that only those scenes with less than or equal to 70% cloud cover will be considered as a candidate for compositing. +The scoring function Max cloud cover in scene indicates the maximum percentage of cloud cover in an image that will be accepted by the user in the BAP image compositing process. Defining a value of 70% implies that only those scenes with less than or equal to 70% cloud cover will be considered as a candidate for compositing. -The Distance to clouds and cloud shadows scoring function enables the user to exclude those pixels identified to contain clouds and shadows by the QA mask from the generated BAP, as well as decreasing a pixel’s score if the pixel is within a specified proximity of a cloud or cloud shadow. +The Distance to clouds and cloud shadows scoring function enables the user to exclude those pixels identified to contain clouds and shadows by the QA mask from the generated BAP, as well as decreasing a pixel’s score if the pixel is within a specified proximity of a cloud or cloud shadow. -The Atmospheric opacity scoring function ranks pixels based on their atmospheric opacity values, which are indicative of hazy imagery. Pixels with opacity values that exceed a defined haze expectation (Max opacity) are excluded. Pixels with opacity values lower than a defined value (Min opacity) get the maximum score. Pixels with values in between these limits are scored following the functions defined by Griffiths et al. (2013). This scoring function is available only for Landsat 5 TM and Landsat 7 ETM+ imagery, which provides the opacity attribute in the image metadata file. +The Atmospheric opacity scoring function ranks pixels based on their atmospheric opacity values, which are indicative of hazy imagery. Pixels with opacity values that exceed a defined haze expectation (Max opacity) are excluded. Pixels with opacity values lower than a defined value (Min opacity) get the maximum score. Pixels with values in between these limits are scored following the functions defined by Griffiths et al. (2013). This scoring function is available only for Landsat 5 TM and Landsat 7 ETM+ imagery, which provides the opacity attribute in the image metadata file. -Finally, there is a Landsat 7 ETM+ SLC-off penalty scoring function that de-emphasizes images acquired following the ETM+ SLC-off malfunction in 2003. The aim of this scoring element is to ensure that TM or OLI data, which do not have stripes, take precedence over ETM+ when using dates after the SLC failure. This allows users to avoid the inclusion of multiple discontinuous small portions of images being used to produce the BAP image composites, thus reducing the spatial variability of the spectral data. The penalty applied to SLC-off imagery is defined directly proportional to the overall score. A large score reduces the chance that SLC-off imagery will be used in the composite. A value of 1 prevents SLC-off imagery from being used. +Finally, there is a Landsat 7 ETM+ SLC-off penalty scoring function that de-emphasizes images acquired following the ETM+ SLC-off malfunction in 2003. The aim of this scoring element is to ensure that TM or OLI data, which do not have stripes, take precedence over ETM+ when using dates after the SLC failure. This allows users to avoid the inclusion of multiple discontinuous small portions of images being used to produce the BAP image composites, thus reducing the spatial variability of the spectral data. The penalty applied to SLC-off imagery is defined directly proportional to the overall score. A large score reduces the chance that SLC-off imagery will be used in the composite. A value of 1 prevents SLC-off imagery from being used. -By default, the GEE-BAP application produces image composites using all the visible bands. The Spectral index option enables the user to produce selected spectral indices from the resulting BAP image composites. Available spectral indices include: Normalized Difference Vegetation Index (NDVI, Fig. F4.3.11), Enhanced Vegetation Index (EVI), and Normalized Burn Ratio (NBR), as well as several indices derived from the Tasseled Cap transformation: Wetness (TCW), Greenness (TCG), Brightness (TCB), and Angle (TCA). Composited indices are able to be downloaded as well as viewed on the map. +By default, the GEE-BAP application produces image composites using all the visible bands. The Spectral index option enables the user to produce selected spectral indices from the resulting BAP image composites. Available spectral indices include: Normalized Difference Vegetation Index (NDVI, Fig. F4.3.11), Enhanced Vegetation Index (EVI), and Normalized Burn Ratio (NBR), as well as several indices derived from the Tasseled Cap transformation: Wetness (TCW), Greenness (TCG), Brightness (TCB), and Angle (TCA). Composited indices are able to be downloaded as well as viewed on the map. -![](F4/image60.png) +![Fig. F4.3.11 Example of a global BAP image composite showing NDVI values generated using the GEE-BAP user interface](F4/image60.png) -Fig. F4.3.11 Example of a global BAP image composite showing NDVI values generated using the GEE-BAP user interface GEE-BAP functions can be accessed programmatically, including pixel scoring parameters, as well as BAP image compositing (BAP), de-spiking (despikeCollection), data-gap infilling (infill), and displaying (ShowCollection) functions. The following code sets the scoring parameter values, then generates and displays the compositing results (Fig. F4.3.12) for a BAP composite that is de-spiked, with data gaps infilled using temporal interpolation. Copy and paste the code below into a new script. +```js // Define required parameters. -var targetDay = '06-01'; -var daysRange = 75; -var cloudsTh = 70; -var SLCoffPenalty = 0.7; -var opacityScoreMin = 0.2; -var opacityScoreMax = 0.3; -var cloudDistMax = 1500; -var despikeTh = 0.65; -var despikeNbands = 3; -var startYear = 2015; -var endYear = 2017; +var targetDay = '06-01'; +var daysRange = 75; +var cloudsTh = 70; +var SLCoffPenalty = 0.7; +var opacityScoreMin = 0.2; +var opacityScoreMax = 0.3; +var cloudDistMax = 1500; +var despikeTh = 0.65; +var despikeNbands = 3; +var startYear = 2015; +var endYear = 2017; // Define study area. -var worldCountries = ee.FeatureCollection('USDOS/LSIB_SIMPLE/2017'); -var colombia = worldCountries.filter(ee.Filter.eq('country_na',   'Colombia')); +var worldCountries = ee.FeatureCollection('USDOS/LSIB_SIMPLE/2017'); +var colombia = worldCountries.filter(ee.Filter.eq('country_na', 'Colombia')); // Load the bap library. -var library = require('users/sfrancini/bap:library'); +var library = require('users/sfrancini/bap:library'); // Calculate BAP. -var BAPCS = library.BAP(null, targetDay, daysRange, cloudsTh, -   SLCoffPenalty, opacityScoreMin, opacityScoreMax, cloudDistMax); +var BAPCS = library.BAP(null, targetDay, daysRange, cloudsTh, + SLCoffPenalty, opacityScoreMin, opacityScoreMax, cloudDistMax); // Despike the collection. -BAPCS = library.despikeCollection(despikeTh, despikeNbands, BAPCS,   1984, 2021, true); +BAPCS = library.despikeCollection(despikeTh, despikeNbands, BAPCS, 1984, 2021, true); // Infill datagaps. BAPCS = library.infill(BAPCS, 1984, 2021, false, true); // Visualize the image. Map.centerObject(colombia, 5); -library.ShowCollection(BAPCS, startYear, endYear, colombia, false,   null); +library.ShowCollection(BAPCS, startYear, endYear, colombia, false, null); library.AddSLider(startYear, endYear); -![](F4/image11.png) +``` +![Fig. F4.3.12 Outcome of the compositing code](F4/image11.png) -Fig. F4.3.12 Outcome of the compositing code -::: {.callout-note} -Code Checkpoint F43d. The book’s repository contains a script that shows what your code should look like at this point. +:::{.callout-note} +Code Checkpoint F43d. The book’s repository contains a script that shows what your code should look like at this point. ::: ## Synthesis {.unnumbered} -Assignment 1. Create composites for other cloudy regions or less cloudy regions. For example, change the country variable to 'Cambodia' or 'Mozambique'. Are more gaps present in the resulting composite? Can you change the compositing rules to improve this (using Acquisition day of year and Day range)? Different regions of the Earth have different cloud seasonal patterns, so the most appropriate date windows to acquire cloud-free composites will change depending on location. Also be aware that the larger the country, the longer it will take to generate the composite. +Assignment 1. Create composites for other cloudy regions or less cloudy regions. For example, change the country variable to 'Cambodia' or 'Mozambique'. Are more gaps present in the resulting composite? Can you change the compositing rules to improve this (using Acquisition day of year and Day range)? Different regions of the Earth have different cloud seasonal patterns, so the most appropriate date windows to acquire cloud-free composites will change depending on location. Also be aware that the larger the country, the longer it will take to generate the composite. -Assignment 2. Similarly, try creating composites for the wet and dry seasons of a region separately. Compare the two composites. Are some features brighter or darker? Is there evidence of drying of vegetation, such as related to leaf loss or reduction in herbaceous ground vegetation? +Assignment 2. Similarly, try creating composites for the wet and dry seasons of a region separately. Compare the two composites. Are some features brighter or darker? Is there evidence of drying of vegetation, such as related to leaf loss or reduction in herbaceous ground vegetation? Assignment 3. Test different cloud threshold values and see if you can find an optimal threshold that balances data gaps against area coverage for your particular target date. ## Conclusion {.unnumbered} -We cannot monitor what we cannot see. Image compositing algorithms provide robust and transparent tools to address issues with clouds, cloud shadows, haze, and smoke in remotely sensed images derived from optical satellite data, and expand data availability for remote sensing applications. The tools and approaches described here should provide you with some useful strategies to aid in mitigating the presence of cloud cover in your data. Note that the quality of image outcomes is a function of the quality of cloud masking routines applied to the source data to generate the various flags that are used in the scoring functions described herein. Different compositing parameters can be used to represent a given location as a function of conditions that are present at a given point in time and the information needs of the end user. Tuning or optimization of compositing parameters is possible (and recommended) to ensure best capture of the physical conditions of interest. +We cannot monitor what we cannot see. Image compositing algorithms provide robust and transparent tools to address issues with clouds, cloud shadows, haze, and smoke in remotely sensed images derived from optical satellite data, and expand data availability for remote sensing applications. The tools and approaches described here should provide you with some useful strategies to aid in mitigating the presence of cloud cover in your data. Note that the quality of image outcomes is a function of the quality of cloud masking routines applied to the source data to generate the various flags that are used in the scoring functions described herein. Different compositing parameters can be used to represent a given location as a function of conditions that are present at a given point in time and the information needs of the end user. Tuning or optimization of compositing parameters is possible (and recommended) to ensure best capture of the physical conditions of interest. ## References {.unnumbered} @@ -1068,7 +1095,7 @@ Zhu Z, Woodcock CE (2012) Object-based cloud and cloud shadow detection in Lands -::: {.callout-tip} +:::{.callout-tip} # Chapter Information ## Author {.unlisted .unnumbered} @@ -1082,14 +1109,14 @@ Karis Tenneson, John Dilger, Crystal Wespestad, Brian Zutta, Andréa P Nicolau, ## Overview {.unlisted .unnumbered} -This chapter introduces change detection mapping. It will teach you how to make a two-date land cover change map using image differencing and threshold-based classification. You will use what you have learned so far in this book to produce a map highlighting changes in the land cover between two time steps. You will first explore differences between the two images extracted from these time steps by creating a difference layer. You will then learn how to directly classify change based on the information in both of your images. +This chapter introduces change detection mapping. It will teach you how to make a two-date land cover change map using image differencing and threshold-based classification. You will use what you have learned so far in this book to produce a map highlighting changes in the land cover between two time steps. You will first explore differences between the two images extracted from these time steps by creating a difference layer. You will then learn how to directly classify change based on the information in both of your images. ## Learning Outcomes {.unlisted .unnumbered} * Creating and exploring how to read a false-color cloud-free Landsat composite image -* Calculating the Normalized Burn Ratio (NBR) index.   -* Creating a two-image difference to help locate areas of change. +* Calculating the Normalized Burn Ratio (NBR) index. +* Creating a two-image difference to help locate areas of change. * Producing a change map and classifying changes using thresholding. ## Assumes you know how to:{.unlisted .unnumbered} @@ -1098,11 +1125,11 @@ This chapter introduces change detection mapping. It will teach you how to make * Import images and image collections, filter, and visualize (Part F1). * Perform basic image analysis: select bands, compute indices, create masks (Part F2). -::: +::: ## Introduction {.unlisted .unnumbered} -Change detection is the process of assessing how landscape conditions are changing by looking at differences in images acquired at different times. This can be used to quantify changes in forest cover—such as those following a volcanic eruption, logging activity, or wildfire—or when crops are harvested (Fig. F4.4.1). For example, using time-series change detection methods, Hansen et al. (2013) quantified annual changes in forest loss and regrowth. Change detection mapping is important for observing, monitoring, and quantifying changes in landscapes over time. Key questions that can be answered using these techniques include identifying whether a change has occurred, measuring the area or the spatial extent of the region undergoing change, characterizing the nature of the change, and measuring the pattern (configuration or composition) of the change (MacLeod and Congalton 1998). +Change detection is the process of assessing how landscape conditions are changing by looking at differences in images acquired at different times. This can be used to quantify changes in forest cover—such as those following a volcanic eruption, logging activity, or wildfire—or when crops are harvested (Fig. F4.4.1). For example, using time-series change detection methods, Hansen et al. (2013) quantified annual changes in forest loss and regrowth. Change detection mapping is important for observing, monitoring, and quantifying changes in landscapes over time. Key questions that can be answered using these techniques include identifying whether a change has occurred, measuring the area or the spatial extent of the region undergoing change, characterizing the nature of the change, and measuring the pattern (configuration or composition) of the change (MacLeod and Congalton 1998). a) @@ -1118,154 +1145,155 @@ c) d) -![](F4/image87.png) +![Fig. F4.4.1 Before and after images of (a) the eruption of Mount St. Helens in Washington State, USA, in 1980 (before, July 10, 1979; after, September 5, 1980); (b) the Camp Fire in California, USA, in 2018 (before, October 7, 2018; after, March 16, 2019); (c) illegal gold mining in the Madre de Dios region of Peru (before, March 31, 2001; after, August 22, 2020); and (d) shoreline changes in Incheon, South Korea (before, May 29, 1981; after, March 11, 2020)](F4/image87.png) -Fig. F4.4.1 Before and after images of (a) the eruption of Mount St. Helens in Washington State, USA, in 1980 (before, July 10, 1979; after, September 5, 1980); (b) the Camp Fire in California, USA, in 2018 (before, October 7, 2018; after, March 16, 2019); (c) illegal gold mining in the Madre de Dios region of Peru (before, March 31, 2001; after, August 22, 2020); and (d) shoreline changes in Incheon, South Korea (before, May 29, 1981; after, March 11, 2020) Many change detection techniques use the same basic premise: that most changes on the landscape result in spectral values that differ between pre-event and post-event images. The challenge can be to separate the real changes of interest—those due to activities on the landscape—from noise in the spectral signal, which can be caused by seasonal variation and phenology, image misregistration, clouds and shadows, radiometric inconsistencies, variability in illumination (e.g., sun angle, sensor position), and atmospheric effects. -Activities that result in pronounced changes in radiance values for a sufficiently long time period are easier to detect using remote sensing change detection techniques than are subtle or short-lived changes in landscape conditions. Mapping challenges can arise if the change event is short-lived, as these are difficult to capture using satellite instruments that only observe a location every several days. Other types of changes occur so slowly or are so vast that they are not easily detected until they are observed using satellite images gathered over a sufficiently long interval of time. Subtle changes that occur slowly on the landscape may be better suited to more computationally demanding methods, such as time-series analysis. Kennedy et al. (2009) provides a nice overview of the concepts and tradeoffs involved when designing landscape monitoring approaches. Additional summaries of change detection methods and recent advances include Singh (1989), Coppin et al. (2004), Lu et al. (2004), and Woodcock et al. (2020). +Activities that result in pronounced changes in radiance values for a sufficiently long time period are easier to detect using remote sensing change detection techniques than are subtle or short-lived changes in landscape conditions. Mapping challenges can arise if the change event is short-lived, as these are difficult to capture using satellite instruments that only observe a location every several days. Other types of changes occur so slowly or are so vast that they are not easily detected until they are observed using satellite images gathered over a sufficiently long interval of time. Subtle changes that occur slowly on the landscape may be better suited to more computationally demanding methods, such as time-series analysis. Kennedy et al. (2009) provides a nice overview of the concepts and tradeoffs involved when designing landscape monitoring approaches. Additional summaries of change detection methods and recent advances include Singh (1989), Coppin et al. (2004), Lu et al. (2004), and Woodcock et al. (2020). -For land cover changes that occur abruptly over large areas on the landscape and are long-lived, a simple two-date image differencing approach is suitable. Two-date image differencing techniques are long-established methods for identifying changes that produce easily interpretable results (Singh 1989). The process typically involves four steps: (1) image selection and preprocessing; (2) data transformation, such as calculating the difference between indices of interest (e.g., the Normalized Difference Vegetation Index (NDVI)) in the pre-event and post-event images; (3) classifying the differenced image(s) using thresholding or supervised classification techniques; and (4) evaluation. +For land cover changes that occur abruptly over large areas on the landscape and are long-lived, a simple two-date image differencing approach is suitable. Two-date image differencing techniques are long-established methods for identifying changes that produce easily interpretable results (Singh 1989). The process typically involves four steps: (1) image selection and preprocessing; (2) data transformation, such as calculating the difference between indices of interest (e.g., the Normalized Difference Vegetation Index (NDVI)) in the pre-event and post-event images; (3) classifying the differenced image(s) using thresholding or supervised classification techniques; and (4) evaluation. -For the practicum, you will select pre-event and post-event image scenes and investigate the conditions in these images in a false-color composite display. Next, you will calculate the NBR index for each scene and create a difference image using the two NBR maps. Finally, you will apply a threshold to the difference image to establish categories of changed versus stable areas (Fig. F4.4.2). +For the practicum, you will select pre-event and post-event image scenes and investigate the conditions in these images in a false-color composite display. Next, you will calculate the NBR index for each scene and create a difference image using the two NBR maps. Finally, you will apply a threshold to the difference image to establish categories of changed versus stable areas (Fig. F4.4.2). -![](F4/image68.png) +![Fig. F4.4.2 Change detection workflow for this practicum](F4/image68.png) -Fig. F4.4.2 Change detection workflow for this practicum ## Preparing Imagery Before beginning a change detection workflow, image preprocessing is essential. The goal is to ensure that each pixel records the same type of measurement at the same location over time. These steps include multitemporal image registration and radiometric and atmospheric corrections, which are especially important. A lot of this work has been automated and already applied to the images that are available in Earth Engine. Image selection is also important. Selection considerations include finding images with low cloud cover and representing the same phenology (e.g., leaf-on or leaf-off). -The code in the block below accesses the USGS Landsat 8 Level 2, Collection 2, Tier 1 dataset and assigns it to the variable landsat8. To improve readability when working with the Landsat 8 ImageCollection, the code selects bands 2–7 and renames them to band names instead of band numbers. +The code in the block below accesses the USGS Landsat 8 Level 2, Collection 2, Tier 1 dataset and assigns it to the variable landsat8. To improve readability when working with the Landsat 8 ImageCollection, the code selects bands 2–7 and renames them to band names instead of band numbers. -var landsat8 = ee.ImageCollection('LANDSAT/LC08/C02/T1_L2') -   .select( -       ['SR_B2', 'SR_B3', 'SR_B4', 'SR_B5', 'SR_B6', 'SR_B7'], -       ['blue', 'green', 'red', 'nir', 'swir1', 'swir2'] -   ); +var landsat8 = ee.ImageCollection('LANDSAT/LC08/C02/T1_L2') + .select( + ['SR_B2', 'SR_B3', 'SR_B4', 'SR_B5', 'SR_B6', 'SR_B7'], + ['blue', 'green', 'red', 'nir', 'swir1', 'swir2'] + ); -Next, you will split the Landsat 8 ImageCollection into two collections, one for each time period, and apply some filtering and sorting to get an image for each of two time periods. In this example, we know there are few clouds for the months of the analysis; if you’re working in a different area, you may need to apply some cloud masking or mosaicing techniques (see Chap. F4.3). +Next, you will split the Landsat 8 ImageCollection into two collections, one for each time period, and apply some filtering and sorting to get an image for each of two time periods. In this example, we know there are few clouds for the months of the analysis; if you’re working in a different area, you may need to apply some cloud masking or mosaicing techniques (see Chap. F4.3). -The code below does several things. First, it creates a new geometry variable to filter the geographic bounds of the image collections. Next, it creates a new variable for the pre-event image by (1) filtering the collection by the date range of interest (e.g., June 2013), (2) filtering the collection by the geometry, (3) sorting by cloud cover so the first image will have the least cloud cover, and (4) getting the first image from the collection. +The code below does several things. First, it creates a new geometry variable to filter the geographic bounds of the image collections. Next, it creates a new variable for the pre-event image by (1) filtering the collection by the date range of interest (e.g., June 2013), (2) filtering the collection by the geometry, (3) sorting by cloud cover so the first image will have the least cloud cover, and (4) getting the first image from the collection. -Now repeat the previous step, but assign it to a post-event image variable and change the filter date to a period after the pre-event image’s date range (e.g., June 2020).   +Now repeat the previous step, but assign it to a post-event image variable and change the filter date to a period after the pre-event image’s date range (e.g., June 2020). -var point = ee.Geometry.Point([-123.64, 42.96]); +var point = ee.Geometry.Point([-123.64, 42.96]); Map.centerObject(point, 11); -var preImage = landsat8 -   .filterBounds(point) -   .filterDate('2013-06-01', '2013-06-30') -   .sort('CLOUD_COVER', true) -   .first();   var postImage = landsat8 -   .filterBounds(point) -   .filterDate('2020-06-01', '2020-06-30') -   .sort('CLOUD_COVER', true) -   .first(); +var preImage = landsat8 + .filterBounds(point) + .filterDate('2013-06-01', '2013-06-30') + .sort('CLOUD_COVER', true) + .first(); var postImage = landsat8 + .filterBounds(point) + .filterDate('2020-06-01', '2020-06-30') + .sort('CLOUD_COVER', true) + .first(); ## Creating False-Color Composites Before running any sort of change detection analysis, it is useful to first visualize your input images to get a sense of the landscape, visually inspect where changes might occur, and identify any problems in the inputs before moving further. As described in Chap. F1.1, false-color composites draw bands from multispectral sensors in the red, green, and blue channels in ways that are designed to illustrate contrast in imagery. Below, you will produce a false-color composite using SWIR-2 in the red channel, NIR in the green channel, and Red in the blue channel (Fig. F4.4.3). -Following the format in the code block below, first create a variable visParam to hold the display parameters, selecting the SWIR-2, NIR, and red bands, with values drawn that are between 7750 and 22200. Next, add the pre-event and post-event images to the map and click Run. Click and drag the opacity slider on the post-event image layer back and forth to view the changes between your two images. +Following the format in the code block below, first create a variable visParam to hold the display parameters, selecting the SWIR-2, NIR, and red bands, with values drawn that are between 7750 and 22200. Next, add the pre-event and post-event images to the map and click Run. Click and drag the opacity slider on the post-event image layer back and forth to view the changes between your two images. -var visParam = {   'bands': ['swir2', 'nir', 'red'],   'min': 7750,   'max': 22200 +var visParam = { 'bands': ['swir2', 'nir', 'red'], 'min': 7750, 'max': 22200 }; Map.addLayer(preImage, visParam, 'pre'); Map.addLayer(postImage, visParam, 'post'); -![](F4/image31.png) +![Fig. F4.4.3 False-color composite using SWIR2, NIR, and red. Vegetation shows up vividly in the green channel due to vegetation being highly reflective in the NIR band. Shades of green can be indicative of vegetation density; water typically shows up as black to dark blue; and burned or barren areas show up as brown.](F4/image31.png) -Fig. F4.4.3 False-color composite using SWIR2, NIR, and red. Vegetation shows up vividly in the green channel due to vegetation being highly reflective in the NIR band. Shades of green can be indicative of vegetation density; water typically shows up as black to dark blue; and burned or barren areas show up as brown. ## Calculating NBR -The next step is data transformation, such as calculating NBR. The advantage of using these techniques is that the data, along with the noise inherent in the data, have been reduced in order to simplify a comparison between two images. Image differencing is done by subtracting the spectral value of the first-date image from that of the second-date image, pixel by pixel (Fig. F4.4.2). Two-date image differencing can be used with a single band or with spectral indices, depending on the application. Identifying the correct band or index to identify change and finding the correct thresholds to classify it are critical to producing meaningful results. Working with indices known to highlight the land cover conditions before and after a change event of interest is a good starting point. For example, the Normalized Difference Water Index would be good for mapping water level changes during flooding events; the NBR is good at detecting soil brightness; and the NDVI can be used for tracking changes in vegetation (although this index does saturate quickly). In some cases, using derived band combinations that have been customized to represent the phenomenon of interest is suggested, such as using the Normalized Difference Fraction Index to monitor forest degradation (see Chap. A3.4). +The next step is data transformation, such as calculating NBR. The advantage of using these techniques is that the data, along with the noise inherent in the data, have been reduced in order to simplify a comparison between two images. Image differencing is done by subtracting the spectral value of the first-date image from that of the second-date image, pixel by pixel (Fig. F4.4.2). Two-date image differencing can be used with a single band or with spectral indices, depending on the application. Identifying the correct band or index to identify change and finding the correct thresholds to classify it are critical to producing meaningful results. Working with indices known to highlight the land cover conditions before and after a change event of interest is a good starting point. For example, the Normalized Difference Water Index would be good for mapping water level changes during flooding events; the NBR is good at detecting soil brightness; and the NDVI can be used for tracking changes in vegetation (although this index does saturate quickly). In some cases, using derived band combinations that have been customized to represent the phenomenon of interest is suggested, such as using the Normalized Difference Fraction Index to monitor forest degradation (see Chap. A3.4). -Examine changes to the landscape caused by fires using NBR, which measures the severity of fires using the equation (NIR − SWIR) / (NIR + SWIR). These bands were chosen because they respond most strongly to the specific changes in forests caused by fire. This type of equation, a difference of variables divided by their sum, is referred to as a normalized difference equation (see Chap. F2.0). The resulting value will always fall between −1 and 1. NBR is useful for determining whether a fire recently occurred and caused damage to the vegetation, but it is not designed to detect other types of land cover changes especially well. +Examine changes to the landscape caused by fires using NBR, which measures the severity of fires using the equation (NIR − SWIR) / (NIR + SWIR). These bands were chosen because they respond most strongly to the specific changes in forests caused by fire. This type of equation, a difference of variables divided by their sum, is referred to as a normalized difference equation (see Chap. F2.0). The resulting value will always fall between −1 and 1. NBR is useful for determining whether a fire recently occurred and caused damage to the vegetation, but it is not designed to detect other types of land cover changes especially well. -First, calculate the NBR for each time period using the built-in normalized difference function. For Landsat 8, be sure to utilize the NIR and SWIR2 bands to calculate NBR. Then, rename each image band with the built-in rename function. +First, calculate the NBR for each time period using the built-in normalized difference function. For Landsat 8, be sure to utilize the NIR and SWIR2 bands to calculate NBR. Then, rename each image band with the built-in rename function. +```js // Calculate NBR. -var nbrPre = preImage.normalizedDifference(['nir', 'swir2']) -   .rename('nbr_pre'); -var nbrPost = postImage.normalizedDifference(['nir', 'swir2']) -   .rename('nbr_post'); +var nbrPre = preImage.normalizedDifference(['nir', 'swir2']) + .rename('nbr_pre'); +var nbrPost = postImage.normalizedDifference(['nir', 'swir2']) + .rename('nbr_post'); -::: {.callout-note} -Code Checkpoint F44a. The book’s repository contains a script that shows what your code should look like at this point. +``` +:::{.callout-note} +Code Checkpoint F44a. The book’s repository contains a script that shows what your code should look like at this point. ::: ## Single Date Transformation Next, we will examine the changes that have occurred, as seen when comparing two specific dates in time. -Subtract the pre-event image from the post-event image using the subtract function. Add the two-date change image to the map with the specialized Fabio Crameri batlow color ramp (Crameri et al. 2020). This color ramp is an example of a color combination specifically designed to be readable by colorblind and color-deficient viewers. Being cognizant of your cartographic choices is an important part of making a good change map. +Subtract the pre-event image from the post-event image using the subtract function. Add the two-date change image to the map with the specialized Fabio Crameri batlow color ramp (Crameri et al. 2020). This color ramp is an example of a color combination specifically designed to be readable by colorblind and color-deficient viewers. Being cognizant of your cartographic choices is an important part of making a good change map. +```js // 2-date change. -var diff = nbrPost.subtract(nbrPre).rename('change'); +var diff = nbrPost.subtract(nbrPre).rename('change'); -var palette = [   '011959', '0E365E', '1D5561', '3E6C55', '687B3E',   '9B882E', 'D59448', 'F9A380', 'FDB7BD', 'FACCFA' +var palette = [ '011959', '0E365E', '1D5561', '3E6C55', '687B3E', '9B882E', 'D59448', 'F9A380', 'FDB7BD', 'FACCFA' ]; -var visParams = { -   palette: palette, -   min: -0.2, -   max: 0.2 +var visParams = { + palette: palette, + min: -0.2, + max: 0.2 }; Map.addLayer(diff, visParams, 'change'); -Question 1. Try to interpret the resulting image before reading on. What patterns of change can you identify? Can you find areas that look like vegetation loss or gain? +``` +Question 1. Try to interpret the resulting image before reading on. What patterns of change can you identify? Can you find areas that look like vegetation loss or gain? -The color ramp has dark blues for the lowest values, greens and oranges in the midrange, and pink for the highest values. We used nbrPre subtracted from nbrPost to identify changes in each pixel. Since NBR values are higher when vegetation is present, areas that are negative in the change image will represent pixels that were higher in the nbrPre image than in the nbrPost image. Conversely, positive differences mean that an area gained vegetation (Fig. F4.4.4).  +The color ramp has dark blues for the lowest values, greens and oranges in the midrange, and pink for the highest values. We used nbrPre subtracted from nbrPost to identify changes in each pixel. Since NBR values are higher when vegetation is present, areas that are negative in the change image will represent pixels that were higher in the nbrPre image than in the nbrPost image. Conversely, positive differences mean that an area gained vegetation (Fig. F4.4.4). -a)                                            b)                                           c) +a) b) c) -![](F4/image4.png) +![Fig. F4.4.4 (a) Two-date NBR difference; (b) pre-event image (June 2013) false-color composite; (c) post-event image (June 2020) false-color composite. In the change map (a), areas on the lower range of values (blue) depict areas where vegetation has been negatively affected, and areas on the higher range of values (pink) depict areas where there has been vegetation gain; the green/orange areas have experienced little change. In the pre-event and post-event images (b and c), the green areas indicate vegetation, while the brown regions are barren ground.](F4/image4.png) -Fig. F4.4.4 (a) Two-date NBR difference; (b) pre-event image (June 2013) false-color composite; (c) post-event image (June 2020) false-color composite. In the change map (a), areas on the lower range of values (blue) depict areas where vegetation has been negatively affected, and areas on the higher range of values (pink) depict areas where there has been vegetation gain; the green/orange areas have experienced little change. In the pre-event and post-event images (b and c), the green areas indicate vegetation, while the brown regions are barren ground. ## Classifying Change -Once the images have been transformed and differenced to highlight areas undergoing change, the next step is image classification into a thematic map consisting of stable and change classes. This can be done rather simply by thresholding the change layer, or by using classification techniques such as machine learning algorithms. One challenge of working with simple thresholding of the difference layer is knowing how to select a suitable threshold to partition changed areas from stable classes. On the other hand, classification techniques using machine learning algorithms partition the landscape using examples of reference data that you provide to train the classifier. This may or may not yield better results, but does require additional work to collect reference data and train the classifier. In the end, resources, timing, and the patterns of the phenomenon you are trying to map will determine which approach is suitable—or perhaps the activity you are trying to track requires something more advanced, such as a time-series approach that uses more than two dates of imagery. +Once the images have been transformed and differenced to highlight areas undergoing change, the next step is image classification into a thematic map consisting of stable and change classes. This can be done rather simply by thresholding the change layer, or by using classification techniques such as machine learning algorithms. One challenge of working with simple thresholding of the difference layer is knowing how to select a suitable threshold to partition changed areas from stable classes. On the other hand, classification techniques using machine learning algorithms partition the landscape using examples of reference data that you provide to train the classifier. This may or may not yield better results, but does require additional work to collect reference data and train the classifier. In the end, resources, timing, and the patterns of the phenomenon you are trying to map will determine which approach is suitable—or perhaps the activity you are trying to track requires something more advanced, such as a time-series approach that uses more than two dates of imagery. For this chapter, we will classify our image into categories using a simple, manual thresholding method, meaning we will decide the optimal values for when a pixel will be considered change or no-change in the image. Finding the ideal value is a considerable task and will be unique to each use case and set of inputs (e.g., the threshold values for a SWIR2 single-band change would be different from the thresholds for NDVI). For a look at a more advanced method of thresholding, check out the thresholding methods in Chap. A2.3. -First, you will define two variables for the threshold values for gain and loss. Next, create a new image with a constant value of 0. This will be the basis of our classification. Reclassify the new image using the where function. Classify loss areas as 2 where the difference image is less than or equal to the loss threshold value. Reclassify gain areas to 1 where the difference image is greater than or equal to the gain threshold value. Finally, mask the image by itself and add the classified image to the map (Fig. F4.4.5). Note: It is not necessary to self-mask the image, and in many cases you might be just as interested in areas that did not change as you are in areas that did. +First, you will define two variables for the threshold values for gain and loss. Next, create a new image with a constant value of 0. This will be the basis of our classification. Reclassify the new image using the where function. Classify loss areas as 2 where the difference image is less than or equal to the loss threshold value. Reclassify gain areas to 1 where the difference image is greater than or equal to the gain threshold value. Finally, mask the image by itself and add the classified image to the map (Fig. F4.4.5). Note: It is not necessary to self-mask the image, and in many cases you might be just as interested in areas that did not change as you are in areas that did. +```js // Classify change -var thresholdGain = 0.10; -var thresholdLoss = -0.10; +var thresholdGain = 0.10; +var thresholdLoss = -0.10; -var diffClassified = ee.Image(0); +var diffClassified = ee.Image(0); diffClassified = diffClassified.where(diff.lte(thresholdLoss), 2); diffClassified = diffClassified.where(diff.gte(thresholdGain), 1); -var changeVis = { -   palette: 'fcffc8,2659eb,fa1373', -   min: 0, -   max: 2 +var changeVis = { + palette: 'fcffc8,2659eb,fa1373', + min: 0, + max: 2 }; Map.addLayer(diffClassified.selfMask(), -   changeVis,   'change classified by threshold'); + changeVis, 'change classified by threshold'); +``` a) ![](F4/image77.png)![](F4/image77.png) b) -![](F4/image17.png)![](F4/image17.png) +![Fig. F4.4.5 (a) Change detection in timber forests of southern Oregon, including maps of the (left to right) pre-event false-color composite, post-event false-color composite, difference image, and classified change using NBR; (b) the same map types for an example of change caused by fire in southern Oregon. The false-color maps highlight vegetation in green and barren ground in brown. The difference images show NBR gain in pink to NBR loss in blue. The classified change images show NBR gain in blue and NBR loss in red.](F4/image17.png)![Fig. F4.4.5 (a) Change detection in timber forests of southern Oregon, including maps of the (left to right) pre-event false-color composite, post-event false-color composite, difference image, and classified change using NBR; (b) the same map types for an example of change caused by fire in southern Oregon. The false-color maps highlight vegetation in green and barren ground in brown. The difference images show NBR gain in pink to NBR loss in blue. The classified change images show NBR gain in blue and NBR loss in red.](F4/image17.png) -Fig. F4.4.5 (a) Change detection in timber forests of southern Oregon, including maps of the (left to right) pre-event false-color composite, post-event false-color composite, difference image, and classified change using NBR; (b) the same map types for an example of change caused by fire in southern Oregon. The false-color maps highlight vegetation in green and barren ground in brown. The difference images show NBR gain in pink to NBR loss in blue. The classified change images show NBR gain in blue and NBR loss in red. -Chapters F4.5 through F4.9 present more-advanced change detection algorithms that go beyond differencing and thresholding between two images, instead allowing you to analyze changes indicated across several images as a time series. +Chapters F4.5 through F4.9 present more-advanced change detection algorithms that go beyond differencing and thresholding between two images, instead allowing you to analyze changes indicated across several images as a time series. -::: {.callout-note} -Code Checkpoint F44b. The book’s repository contains a script that shows what your code should look like at this point. +:::{.callout-note} +Code Checkpoint F44b. The book’s repository contains a script that shows what your code should look like at this point. ::: ## Synthesis {.unnumbered} @@ -1273,9 +1301,9 @@ Evaluating any maps you create, including change detection maps, is essential to Assignment 1. Try using a different index, such as NDVI or a Tasseled Cap Transformation, to run the change detection steps, and compare the results with those obtained from using NBR. -Assignment 2. Experiment with adjusting the thresholdLoss and thresholdGain values. +Assignment 2. Experiment with adjusting the thresholdLoss and thresholdGain values. -Assignment 3. Use what you have learned in the classification chapter (Chap. F2.1) to run a supervised classification on the difference layer (or layers, if you have created additional ones). Hint: To complete a supervised classification, you would need reference examples of both the stable and change classes of interest to train the classifier. +Assignment 3. Use what you have learned in the classification chapter (Chap. F2.1) to run a supervised classification on the difference layer (or layers, if you have created additional ones). Hint: To complete a supervised classification, you would need reference examples of both the stable and change classes of interest to train the classifier. Assignment 4. Think about how things like clouds and cloud shadows could affect the results of change detection. What do you think the two-date differencing method would pick up for images in the same year in different seasons? @@ -1301,7 +1329,7 @@ Lu D, Mausel P, Brondízio E, Moran E (2004) Change detection techniques. Int J Macleod RD, Congalton RG (1998) A quantitative comparison of change-detection algorithms for monitoring eelgrass from remotely sensed data. Photogramm Eng Remote Sensing 64:207–216 -Singh A (1989) Digital change detection techniques using remotely-sensed data. Int J  Remote Sens 10:989–1003. https://doi.org/10.1080/01431168908903939 +Singh A (1989) Digital change detection techniques using remotely-sensed data. Int J Remote Sens 10:989–1003. https://doi.org/10.1080/01431168908903939 Stehman SV, Czaplewski RL (1998) Design and analysis for thematic map accuracy assessment: Fundamental principles. Remote Sens Environ 64:331–344. https://doi.org/10.1016/S0034-4257(98)00010-8 @@ -1315,14 +1343,14 @@ Woodcock CE, Loveland TR, Herold M, Bauer ME (2020) Transitioning from change de -::: {.callout-tip} +:::{.callout-tip} # Chapter Information ## Author {.unlisted .unnumbered} -Robert Kennedy, Justin Braaten, Peter Clary +Robert Kennedy, Justin Braaten, Peter Clary @@ -1336,7 +1364,7 @@ Time-series analysis of change can be achieved by fitting the entire spectral tr * Evaluating yearly time-series spectral values to distinguish between true change and artifacts. * Recognizing disturbance and growth signals in the time series of annual spectral values for individual pixels. -* Interpreting change segments and translating them to maps. +* Interpreting change segments and translating them to maps. * Applying parameters in a graphical user interface to create disturbance maps in forests. ## Assumes you know how to:{.unlisted .unnumbered} @@ -1345,7 +1373,7 @@ Time-series analysis of change can be achieved by fitting the entire spectral tr * Calculate and interpret vegetation indices (Chap. F2.0) * Interpret bands and indices in terms of land surface characteristics (Chap. F2.0). -::: +::: ## Introduction {.unlisted .unnumbered} @@ -1356,170 +1384,162 @@ In this lab, we use the LandTrendr time-series algorithms to map change. The Lan For this lab, we will use a graphical user interface (GUI) to teach the concepts of LandTrendr. -::: {.callout-note} -Code Checkpoint F45a. The book’s repository contains information about accessing the LandTrendr interface. +:::{.callout-note} +Code Checkpoint F45a. The book’s repository contains information about accessing the LandTrendr interface. ::: ## Pixel Time Series -When working with LandTrendr for the first time in your area, there are two questions you must address.   +When working with LandTrendr for the first time in your area, there are two questions you must address. -First, is the change of interest detectable in the spectral reflectance record? If the change you are interested in does not leave a pattern in the spectral reflectance record, then an algorithm will not be able to find it.   +First, is the change of interest detectable in the spectral reflectance record? If the change you are interested in does not leave a pattern in the spectral reflectance record, then an algorithm will not be able to find it. Second, can you identify fitting parameters that allow the algorithm to capture that record? Time series algorithms apply rules to a temporal sequence of spectral values in a pixel, and simplify the many observations into more digestible forms, such as the linear segments we will work with using LandTrendr. The algorithms that do the simplification are often guided by parameters that control the way the algorithm does its job. -The best way to begin assessing these questions is to look at the time series of individual pixels. In Earth Engine, open and run the script that generates the GUI we have developed to easily deploy the LandTrendr algorithms. Run the script, and you should see an interface that looks like the one shown in Fig. 4.5.1. +The best way to begin assessing these questions is to look at the time series of individual pixels. In Earth Engine, open and run the script that generates the GUI we have developed to easily deploy the LandTrendr algorithms. Run the script, and you should see an interface that looks like the one shown in Fig. 4.5.1. -![](F4/image29.png) +![Fig. 4.5.1 The LandTrendr GUI interface, with the control panel on the left, the Map panel in the center, and the reporting panel on the right](F4/image29.png) -Fig. 4.5.1 The LandTrendr GUI interface, with the control panel on the left, the  Map panel in the center, and the reporting panel on the right -The LandTrendr GUI consists of three panels: a control panel on the left, a reporting panel on the right, and a Map panel in the center. The control panel is where all of the functionality of the interface resides. There are several modules,and each is accessed by clicking on the double arrow to the right of the title. The  Map panel defaults to a location in Oregon but can be manually moved anywhere in the world. The reporting panel shows messages about how to use functions, as well as providing graphical outputs. +The LandTrendr GUI consists of three panels: a control panel on the left, a reporting panel on the right, and a Map panel in the center. The control panel is where all of the functionality of the interface resides. There are several modules,and each is accessed by clicking on the double arrow to the right of the title. The Map panel defaults to a location in Oregon but can be manually moved anywhere in the world. The reporting panel shows messages about how to use functions, as well as providing graphical outputs. -Next, expand the “Pixel Time Series Options” function. For now, simply use your mouse to click somewhere on the map. Wait a few seconds even though it looks like nothing is happening – be patient!! The GUI has sent information to Earth Engine to run the LandTrendr algorithms at the location you have clicked, and is waiting for the results. Eventually you should see a chart appear in the reporting panel on the right. Fig. 4.5.2 shows what one pixel looks like in an area where the forest burned and began regrowth. Your chart will probably look different. +Next, expand the “Pixel Time Series Options” function. For now, simply use your mouse to click somewhere on the map. Wait a few seconds even though it looks like nothing is happening – be patient!! The GUI has sent information to Earth Engine to run the LandTrendr algorithms at the location you have clicked, and is waiting for the results. Eventually you should see a chart appear in the reporting panel on the right. Fig. 4.5.2 shows what one pixel looks like in an area where the forest burned and began regrowth. Your chart will probably look different. -![](F4/image88.png) +![Fig. 4.5.2 A typical trajectory for a single pixel. The x-axis shows the year, the y-axis the spectral index value, and the title the index chosen. The gray line represents the original spectral values observed by Landsat, and the red line the result of the LandTrendr temporal segmentation algorithms.](F4/image88.png) -Fig. 4.5.2 A typical trajectory for a single pixel. The x-axis shows the year, the y-axis the spectral index value, and the title the index chosen. The gray line represents the original spectral values observed by Landsat, and the red line the result of the LandTrendr temporal segmentation algorithms. The key to success with the LandTrendr algorithm is interpreting these time series. First, let’s examine the components of the chart. The x-axis shows the year of observation. With LandTrendr, only one observation per year is used to describe the history of a pixel; later, we will cover how you control that value. The y-axis shows the spectral value of the index that is chosen. In the default mode, the Normalized Burn Ratio (as described in Chap. F4.4). Note that you also have the ability to pick more indices using the checkboxes on the control panel on the left. Note that we scale floating point (decimal) indices by 1000. Thus, an NBR value of 1.0 would be displayed as 1000. -In the chart area, the thick gray line represents the spectral values observed by the satellite for the period of the year selected for a single 30 m Landsat pixel at the location you have chosen. The red line is the output from the temporal segmentation that is the heart of the LandTrendr algorithms. The title of the chart shows the spectral index, as well as the root-mean-square error of the fit.  +In the chart area, the thick gray line represents the spectral values observed by the satellite for the period of the year selected for a single 30 m Landsat pixel at the location you have chosen. The red line is the output from the temporal segmentation that is the heart of the LandTrendr algorithms. The title of the chart shows the spectral index, as well as the root-mean-square error of the fit. To interpret the time series, first know which way is “up” and “down” for the spectral index you’re interested in. For the NBR, the index goes up in value when there is more vegetation and less soil in a pixel. It goes down when there is less vegetation. For vegetation disturbance monitoring, this is useful. -Next, translate that change into the changes of interest for the change processes you’re interested in. For conifer forest systems, the NBR is useful because it drops precipitously when a disturbance occurs, and it rises as vegetation grows.   +Next, translate that change into the changes of interest for the change processes you’re interested in. For conifer forest systems, the NBR is useful because it drops precipitously when a disturbance occurs, and it rises as vegetation grows. -In the case of Fig. 4.5.2, we interpret the abrupt drop as a disturbance, and the subsequent rise of the index as regrowth or recovery (though not necessarily to the same type of vegetation). +In the case of Fig. 4.5.2, we interpret the abrupt drop as a disturbance, and the subsequent rise of the index as regrowth or recovery (though not necessarily to the same type of vegetation). -![](F4/image71.png) +![Fig. 4.5.3 For the trajectory in Fig. 4.5.2, we can identify a segment capturing disturbance based on its abrupt drop in the NBR index, and the subsequent vegetative recovery](F4/image71.png) -Fig. 4.5.3 For the trajectory in Fig. 4.5.2, we can identify a segment capturing disturbance based on its abrupt drop in the NBR index, and the subsequent vegetative recovery -Tip: LandTrendr is able to accept any index, and advanced users are welcome to use indices of their own design. An important consideration is knowing which direction indicates “recovery” and “disturbance” for the topic you are interested in. The algorithms favor detection of disturbance and can be controlled to constrain how quickly recovery is assumed to occur (see parameters below).   +Tip: LandTrendr is able to accept any index, and advanced users are welcome to use indices of their own design. An important consideration is knowing which direction indicates “recovery” and “disturbance” for the topic you are interested in. The algorithms favor detection of disturbance and can be controlled to constrain how quickly recovery is assumed to occur (see parameters below). For LandTrendr to have any hope of finding the change of interest, that change must be manifested in the gray line showing the original spectral values. If you know that some process is occurring and it is not evident in the gray line, what can you do? One option is to change the index. Any single index is simply one view of the larger spectral space of the Landsat Thematic Mapper sensors. The change you are interested in may cause spectral change in a different direction than that captured with some indices. Try choosing different indices from the list. If you click on different checkboxes and re-submit the pixel, the fits for all of the different indices will appear. -Another option is to change the date range. LandTrendr uses one value per year, but the value that is chosen can be controlled by the user. It’s possible that the change of interest is better identified in some seasons than others. We use a medoid image compositing approach, which picks the best single observation each year from a date range of images in an ImageCollection. In the GUI, you can change the date range of imagery used for compositing in the Image Collection portion of the LandTrendr Options menu (Fig. F4.5.4).   +Another option is to change the date range. LandTrendr uses one value per year, but the value that is chosen can be controlled by the user. It’s possible that the change of interest is better identified in some seasons than others. We use a medoid image compositing approach, which picks the best single observation each year from a date range of images in an ImageCollection. In the GUI, you can change the date range of imagery used for compositing in the Image Collection portion of the LandTrendr Options menu (Fig. F4.5.4). -![](F4/image7.png) +![Fig. 4.5.4 The LandTrendr options menu. Users control the year and date range in the Image Collection section, the index used for temporal segmentation in the middle section, and the parameters controlling the temporal segmentation in the bottom section](F4/image7.png) -Fig. 4.5.4 The LandTrendr options menu. Users control the year and date range in the Image Collection section, the index used for temporal segmentation in the middle section, and the parameters controlling the temporal segmentation in the bottom section -Change the Start Date and End Date to find a time of year when the distinction between cover conditions before and during the change process of interest is greatest. +Change the Start Date and End Date to find a time of year when the distinction between cover conditions before and during the change process of interest is greatest. -There are other considerations to keep in mind. First, seasonality of vegetation, water, or snow often can affect the signal of the change of interest. And because we use an ImageCollection that spans a range of dates, it’s best to choose a date range where there is not likely to be a substantial change in vegetative state from the beginning to the end of the date range. Clouds can be a factor too. Some seasons will have more cloudiness, which can make it difficult to find good images. Often with optical sensors, we are constrained to working with periods where clouds are less prevalent, or using wide date ranges to provide many opportunities for a pixel to be cloud-free. +There are other considerations to keep in mind. First, seasonality of vegetation, water, or snow often can affect the signal of the change of interest. And because we use an ImageCollection that spans a range of dates, it’s best to choose a date range where there is not likely to be a substantial change in vegetative state from the beginning to the end of the date range. Clouds can be a factor too. Some seasons will have more cloudiness, which can make it difficult to find good images. Often with optical sensors, we are constrained to working with periods where clouds are less prevalent, or using wide date ranges to provide many opportunities for a pixel to be cloud-free. It is possible that no combination of index or data range is sensitive to the change of interest. If that is the case, there are two options: try using a different sensor and change detection technique, or accept that the change is not discernible. This can often occur if the change of interest occupies a small portion of a given 30 m by 30 m Landsat pixel, or if the spectral manifestation of the change is so subtle that it is not spectrally separable from non-changed pixels -Even if you as a human can identify the change of interest in the spectral trajectory of the gray line, an algorithm may not be able to similarly track it. To give the algorithm a fighting chance, you need to explore whether different fitting parameters could be used to match the red fitted line with the gray source image line.   +Even if you as a human can identify the change of interest in the spectral trajectory of the gray line, an algorithm may not be able to similarly track it. To give the algorithm a fighting chance, you need to explore whether different fitting parameters could be used to match the red fitted line with the gray source image line. -The overall fitting process includes steps to reduce noise and best identify the underlying signal. The temporal segmentation algorithms are controlled by fitting parameters that are described in detail in [Kennedy et al. (2010)](https://www.google.com/url?q=https://www.zotero.org/google-docs/?Siyubi&sa=D&source=editors&ust=1671458868286041&usg=AOvVaw1IKLEXLb8EIQLL40vCpog-). You adjust these parameters using the Fitting Parameters block of the LandTrendr Options menu. Below is a brief overview of what values are often useful, but these will likely change as you use different spectral indices. +The overall fitting process includes steps to reduce noise and best identify the underlying signal. The temporal segmentation algorithms are controlled by fitting parameters that are described in detail in [Kennedy et al. (2010)](https://www.google.com/url?q=https://www.zotero.org/google-docs/?Siyubi&sa=D&source=editors&ust=1671458868286041&usg=AOvVaw1IKLEXLb8EIQLL40vCpog-). You adjust these parameters using the Fitting Parameters block of the LandTrendr Options menu. Below is a brief overview of what values are often useful, but these will likely change as you use different spectral indices. -First, the minimum observations needed criterion is used to evaluate whether a given trajectory has enough unfiltered (i.e., clear observation) years to run the fitting. We suggest leaving this at the default of 6. +First, the minimum observations needed criterion is used to evaluate whether a given trajectory has enough unfiltered (i.e., clear observation) years to run the fitting. We suggest leaving this at the default of 6. -The segmentation begins with a noise-dampening step to remove spikes that could be caused by unfiltered clouds or shadows. The spike threshold parameter controls the degree of filtering. A value of 1.0 corresponds to no filtering, and lower values corresponding to more severe filtering. We suggest leaving this at 0.9; if changed, a range from 0.7 to 1.0 is appropriate. +The segmentation begins with a noise-dampening step to remove spikes that could be caused by unfiltered clouds or shadows. The spike threshold parameter controls the degree of filtering. A value of 1.0 corresponds to no filtering, and lower values corresponding to more severe filtering. We suggest leaving this at 0.9; if changed, a range from 0.7 to 1.0 is appropriate. -The next step is finding vertices. This begins with the start and end year as vertex years, progressively adding candidate vertex years based on deviation from linear fits. To avoid getting an overabundance of vertex years initially found using this method, we suggest leaving the vertex count overshoot at a value of 3. A second set of algorithms uses deflection angle to cull back this overabundance to a set number of maximum candidate vertex years. +The next step is finding vertices. This begins with the start and end year as vertex years, progressively adding candidate vertex years based on deviation from linear fits. To avoid getting an overabundance of vertex years initially found using this method, we suggest leaving the vertex count overshoot at a value of 3. A second set of algorithms uses deflection angle to cull back this overabundance to a set number of maximum candidate vertex years. -That number of vertex years is controlled by the max_segments parameter. As a general rule, your number of segments should be no more than one-third of the total number of likely yearly observations. The years of these vertices (X-values) are then passed to the model-building step. Assuming you are using at least 30 years of the archive, and your area has reasonable availability of images, a value of 8 is a good starting point. +That number of vertex years is controlled by the max_segments parameter. As a general rule, your number of segments should be no more than one-third of the total number of likely yearly observations. The years of these vertices (X-values) are then passed to the model-building step. Assuming you are using at least 30 years of the archive, and your area has reasonable availability of images, a value of 8 is a good starting point. -In the model-building step, straight-line segments are built by fitting Y-values (spectral values) for the periods defined by the vertex years (X-values). The process moves from left to right—early years to late years. Regressions of each subsequent segment are connected to the end of the prior segment. Regressions are also constrained to prevent unrealistic recovery after disturbance, as controlled by the recovery threshold parameter. A lower value indicates greater constraint: a value of 1.0 means the constraint is turned off; a value of 0.25 means that segments that fully recover in faster than four years (4 = 1/0.25) are not permitted. Note: This parameter has strong control on the fitting, and is one of the first to explore when testing parameters.  Additionally, the preventOneYearRecovery will disallow fits that have one-year-duration recovery segments.  This may be useful to prevent overfitting of noisy data in environments where such quick vegetative recovery is not ecologically realistic. +In the model-building step, straight-line segments are built by fitting Y-values (spectral values) for the periods defined by the vertex years (X-values). The process moves from left to right—early years to late years. Regressions of each subsequent segment are connected to the end of the prior segment. Regressions are also constrained to prevent unrealistic recovery after disturbance, as controlled by the recovery threshold parameter. A lower value indicates greater constraint: a value of 1.0 means the constraint is turned off; a value of 0.25 means that segments that fully recover in faster than four years (4 = 1/0.25) are not permitted. Note: This parameter has strong control on the fitting, and is one of the first to explore when testing parameters. Additionally, the preventOneYearRecovery will disallow fits that have one-year-duration recovery segments. This may be useful to prevent overfitting of noisy data in environments where such quick vegetative recovery is not ecologically realistic. -Once a model of the maximum number of segments is found, successively simpler models are made by iteratively removing the least informative vertex. Each model is scored using a pseudo-f statistic, which penalizes models with more segments, to create a pseudo p-value for each model. The p-value threshold parameter is used to identify all fits that are deemed good enough. Start with a value of 0.05, but check to see if the fitted line appears to capture the salient shape and features of the gray source trajectory.  If you see temporal patterns in the gray line that are likely not noise (based on your understanding of the system under study), consider switching the p-value threshold to 0.10 or even 0.15.  +Once a model of the maximum number of segments is found, successively simpler models are made by iteratively removing the least informative vertex. Each model is scored using a pseudo-f statistic, which penalizes models with more segments, to create a pseudo p-value for each model. The p-value threshold parameter is used to identify all fits that are deemed good enough. Start with a value of 0.05, but check to see if the fitted line appears to capture the salient shape and features of the gray source trajectory. If you see temporal patterns in the gray line that are likely not noise (based on your understanding of the system under study), consider switching the p-value threshold to 0.10 or even 0.15. Note: because of temporal autocorrelation, these cannot be interpreted as true f- and p-values, but rather as relative scalars to distinguish goodness of fit among models. If no good models can be found using these criteria based on the p-value parameter set by the user, a second approach is used to solve for the Y-value of all vertex years simultaneously. If no good model is found, then a straight-line mean value model is used. -From the models that pass the p-value threshold, one is chosen as the final fit. It may be the one with the lowest p-value. However, an adjustment is made to allow more complicated models (those with more segments) to be picked even if their p-value is within a defined proportion of the best-scoring model. That proportion is set by the best model proportion parameter. As an example, a best model proportion value of 0.75 would allow a more complicated model to be chosen if its score were greater than 75% that of the best model. +From the models that pass the p-value threshold, one is chosen as the final fit. It may be the one with the lowest p-value. However, an adjustment is made to allow more complicated models (those with more segments) to be picked even if their p-value is within a defined proportion of the best-scoring model. That proportion is set by the best model proportion parameter. As an example, a best model proportion value of 0.75 would allow a more complicated model to be chosen if its score were greater than 75% that of the best model. ## Translating Pixels to Maps -Although the full time series is the best description of each pixel’s “life history,” we typically are interested in the behavior of all of the pixels in our study area. It would be both inefficient to manually visualize all of them and ineffective to try to summarize areas and locations. Thus, we seek to make maps. +Although the full time series is the best description of each pixel’s “life history,” we typically are interested in the behavior of all of the pixels in our study area. It would be both inefficient to manually visualize all of them and ineffective to try to summarize areas and locations. Thus, we seek to make maps. -There are three post-processing steps to convert a segmented trajectory to a map. First, we identify segments of interest; if we are interested in disturbance, we find segments whose spectral change indicates loss. Second, we filter out segments of that type that do not meet criteria of interest. For example, very low magnitude disturbances can occur when the algorithm mistakenly finds a pattern in the random noise of the signal, and thus we do not want to include it. Third, we extract from the segment of interest something about its character to map on a pixel-by-pixel basis: its start year, duration, spectral value, or the value of the spectral change. +There are three post-processing steps to convert a segmented trajectory to a map. First, we identify segments of interest; if we are interested in disturbance, we find segments whose spectral change indicates loss. Second, we filter out segments of that type that do not meet criteria of interest. For example, very low magnitude disturbances can occur when the algorithm mistakenly finds a pattern in the random noise of the signal, and thus we do not want to include it. Third, we extract from the segment of interest something about its character to map on a pixel-by-pixel basis: its start year, duration, spectral value, or the value of the spectral change. -Theory: We’ll start with a single pixel to learn how to Interpret a disturbance pixel time series in terms of the dominant disturbance segment. For the disturbance time series we have used in figures above, we can identify the key parameters of the segment associated with the disturbance. For the example above, we have extracted the actual NBR values of the fitted time series and noted them in a table (Fig. 4.5.5). This is not part of the GUI – it is simply used here to work through the concepts. +Theory: We’ll start with a single pixel to learn how to Interpret a disturbance pixel time series in terms of the dominant disturbance segment. For the disturbance time series we have used in figures above, we can identify the key parameters of the segment associated with the disturbance. For the example above, we have extracted the actual NBR values of the fitted time series and noted them in a table (Fig. 4.5.5). This is not part of the GUI – it is simply used here to work through the concepts. -![](F4/image84.png) +![Fig. 4.5.5 Tracking actual values of fitted trajectories to learn how we focus on quantification of disturbance. Because we know that the NBR index drops when vegetation is lost and soil exposure is increased, we know that a precipitous drop suggests an abrupt loss of vegetation. Although some early segments show very subtle change, only the segment between vertex 4 and 5 shows large-magnitude vegetation loss.](F4/image84.png) -Fig. 4.5.5 Tracking actual values of fitted trajectories to learn how we focus on quantification of disturbance. Because we know that the NBR index drops when vegetation is lost and soil exposure is increased, we know that a precipitous drop suggests an abrupt loss of vegetation. Although some early segments show very subtle change, only the segment between vertex 4 and 5 shows large-magnitude vegetation loss. From the table shown in Fig. 4.5.5, we can infer several key things about this pixel: -* It was likely disturbed between 2006 and 2007. This is because the NBR value drops precipitously in the segment bounded by vertices (breakpoints) in 2006 and 2007.   +* It was likely disturbed between 2006 and 2007. This is because the NBR value drops precipitously in the segment bounded by vertices (breakpoints) in 2006 and 2007. * The magnitude of spectral change was large: 1175 scaled NBR units out of a possible range of 2000 scaled units. * There were small drops in NBR earlier, which may indicate some subtle loss of vegetation over a long period in the pixel. These drops, however, would need to be explored in a separate analysis because of their subtle nature. * The main disturbance had a disturbance duration of just one year. This abruptness combined with the high magnitude suggests a major vegetative disturbance such as a harvest or a fire. * The disturbance was then followed by recovery of vegetation, but not to the level before the disturbance. Note: Ecologists will recognize the growth signal as one of succession, or active revegetation by human intervention. -Following the three post-processing steps noted in the introduction to this section, to map the year of disturbance for this pixel we would first identify the potential disturbance segments as those with negative NBR. Then we would hone in on the disturbance of interest by filtering out potential disturbance segments that are not abrupt and/or of small magnitude. This would leave only the high-magnitude, short-duration segment. For that segment, the first year that we have evidence of disturbance is the first year after the start of the segment. The segment starts in 2006, which means that 2007 is the first year we have such evidence. Thus, we would assign 2007 to this pixel. +Following the three post-processing steps noted in the introduction to this section, to map the year of disturbance for this pixel we would first identify the potential disturbance segments as those with negative NBR. Then we would hone in on the disturbance of interest by filtering out potential disturbance segments that are not abrupt and/or of small magnitude. This would leave only the high-magnitude, short-duration segment. For that segment, the first year that we have evidence of disturbance is the first year after the start of the segment. The segment starts in 2006, which means that 2007 is the first year we have such evidence. Thus, we would assign 2007 to this pixel. If we wanted to map the magnitude of the disturbance, we would follow the same first two steps, but then report for the pixel value the magnitude difference between the starting and ending segment. -The LandTrendr GUI provides a set of tools to easily apply the same logic rules to all pixels of interest and create maps. Click on the Change Filter Options menu. The interface shown in Fig. 4.5.6 appears. +The LandTrendr GUI provides a set of tools to easily apply the same logic rules to all pixels of interest and create maps. Click on the Change Filter Options menu. The interface shown in Fig. 4.5.6 appears. -![](F4/image49.png) +![Fig. 4.5.6 The menu used to post-process disturbance trajectories into maps. Select vegetation change type and sort to hone in on the segment type of interest, then check boxes to apply selective filters to eliminate uninteresting changes. ](F4/image49.png) -Fig. 4.5.6 The menu used to post-process disturbance trajectories into maps. Select vegetation change type and sort to hone in on the segment type of interest, then check boxes to apply selective filters to eliminate uninteresting changes.   -The first two sections are used to identify the segments of interest.   +The first two sections are used to identify the segments of interest. -Select Vegetation Change Type offers the options of gain or loss, which refer to gain or loss of vegetation, with disturbance assumed to be related to loss of vegetation. Note: Advanced users can look in the landtrendr.js library in the “calcindex” function to add new indices with gain and loss defined as they choose. The underlying algorithm is built to find disturbance in indices that increase when disturbance occurs, so indices such as NBR or NDVI need to be multiplied by (−1) before being fed to the LandTrendr algorithm. This is handled in the calcIndex function. +Select Vegetation Change Type offers the options of gain or loss, which refer to gain or loss of vegetation, with disturbance assumed to be related to loss of vegetation. Note: Advanced users can look in the landtrendr.js library in the “calcindex” function to add new indices with gain and loss defined as they choose. The underlying algorithm is built to find disturbance in indices that increase when disturbance occurs, so indices such as NBR or NDVI need to be multiplied by (−1) before being fed to the LandTrendr algorithm. This is handled in the calcIndex function. -Select Vegetation Change Sort offers various options that allow you to choose the segment of interest based on timing or duration. By default, the greatest magnitude disturbance is chosen. +Select Vegetation Change Sort offers various options that allow you to choose the segment of interest based on timing or duration. By default, the greatest magnitude disturbance is chosen. -Each filter (magnitude, duration, etc.) is used to further winnow the possible segments of interest. All other filters are applied at the pixel scale, but Filter by MMU is applied to groups of pixels based on a given minimum mapping unit (MMU). Once all other filters have been defined, some pixels are flagged as being of interest and others are not. The MMU filter looks to see how many connected pixels have been flagged as occurring in the same year, and omits groups smaller in pixel count than the number indicated here (which defaults to 11 pixels, or approximately 1 hectare).   +Each filter (magnitude, duration, etc.) is used to further winnow the possible segments of interest. All other filters are applied at the pixel scale, but Filter by MMU is applied to groups of pixels based on a given minimum mapping unit (MMU). Once all other filters have been defined, some pixels are flagged as being of interest and others are not. The MMU filter looks to see how many connected pixels have been flagged as occurring in the same year, and omits groups smaller in pixel count than the number indicated here (which defaults to 11 pixels, or approximately 1 hectare). -If you’re following along and making changes, or if you’re just using the default location and parameters, click the Add Filtered Disturbance Imagery to add this to the map. You should see something like Fig. 4.5.7. +If you’re following along and making changes, or if you’re just using the default location and parameters, click the Add Filtered Disturbance Imagery to add this to the map. You should see something like Fig. 4.5.7. -![](F4/image16.png) +![Fig. 4.5.7 The basic output from a disturbance mapping exercise ](F4/image16.png) -Fig. 4.5.7 The basic output from a disturbance mapping exercise   There are multiple layers of disturbance added to the map. Use the map layers checkboxes to change which is shown. Magnitude of disturbance, for example, is a map of the delta change between beginning and endpoints of the segments (Fig. 4.5.8). -![](F4/image14.png) +![Fig. 4.5.8 Magnitude of change for the same area](F4/image14.png) -Fig. 4.5.8 Magnitude of change for the same area ## Synthesis {.unnumbered} -In this chapter, you have learned how to work with annual time series to interpret regions of interest. Looking at annual snapshots of the landscape provides three key benefits: (1) the ability to view your area of interest without the clouds and noise that typically obscure single-day views; (2) gauge the amount by which the noise-dampened signal still varies from year to year in response to large-scale forcing mechanisms; and (3) the ability to view the response of landscapes as they recover, sometimes slowly, from disturbance. +In this chapter, you have learned how to work with annual time series to interpret regions of interest. Looking at annual snapshots of the landscape provides three key benefits: (1) the ability to view your area of interest without the clouds and noise that typically obscure single-day views; (2) gauge the amount by which the noise-dampened signal still varies from year to year in response to large-scale forcing mechanisms; and (3) the ability to view the response of landscapes as they recover, sometimes slowly, from disturbance. To learn more about LandTrendr, see the assignments below. -Assignment 1. Find your own change processes of interest. First, navigate the map (zooming and dragging) to an area of the world where you are interested in a change process, and the spectral index that would capture it. Make sure the UI control panel is open to the Pixel Time-Series Options section. Next, click on the map in areas where you know change has occurred, and observe the spectral trajectories in the charts. Then, describe whether the change of interest is detectable in the spectral reflectance record, and what are its characteristics in different parts of the study area. . +Assignment 1. Find your own change processes of interest. First, navigate the map (zooming and dragging) to an area of the world where you are interested in a change process, and the spectral index that would capture it. Make sure the UI control panel is open to the Pixel Time-Series Options section. Next, click on the map in areas where you know change has occurred, and observe the spectral trajectories in the charts. Then, describe whether the change of interest is detectable in the spectral reflectance record, and what are its characteristics in different parts of the study area. . -Assignment 2: Find a pixel in your area of interest that shows a distinctive disturbance process, as you define it for your topic of interest. Adjust date ranges, parameters, etc. using the steps outlined in Section 1 above, and then answer these questions: +Assignment 2: Find a pixel in your area of interest that shows a distinctive disturbance process, as you define it for your topic of interest. Adjust date ranges, parameters, etc. using the steps outlined in Section 1 above, and then answer these questions: -* Question 1. Which index and date range did you use? -* Question 2. Did you need to change fitting parameters to make the algorithm find the disturbance? If so, which ones, and why? -* Question 3. How do you know this is a disturbance? +* Question 1. Which index and date range did you use? +* Question 2. Did you need to change fitting parameters to make the algorithm find the disturbance? If so, which ones, and why? +* Question 3. How do you know this is a disturbance? -Assignment 3. Switch the control panel in the GUI to Change Filter Options, and use the guidance in Section 2 to set parameters and make disturbance maps.   +Assignment 3. Switch the control panel in the GUI to Change Filter Options, and use the guidance in Section 2 to set parameters and make disturbance maps. -* Question 4. Do the disturbance year and magnitude as mapped in the image match with what you would expect from the trajectory itself? -* Question 5. Can you change some of the filters to create a map where your disturbance process is not mapped? If so, what did you change?   -* Question 6. Can you change filters to create a map that includes a different disturbance process, perhaps subtler, longer duration, etc.? Find a pixel and use the “Pixel time series” plotter to look at the time series of those processes. +* Question 4. Do the disturbance year and magnitude as mapped in the image match with what you would expect from the trajectory itself? +* Question 5. Can you change some of the filters to create a map where your disturbance process is not mapped? If so, what did you change? +* Question 6. Can you change filters to create a map that includes a different disturbance process, perhaps subtler, longer duration, etc.? Find a pixel and use the “Pixel time series” plotter to look at the time series of those processes. -Assignment 4:  Return to the Pixel Time-Series Options section of the control panel, and navigate to a pixel in your area of interest that you believe would show a distinctive recovery or growth process, as you define it for your topic of interest. You may want to modify the index, parameters, etc. as covered in Section 1 to adequately capture the growth process with the fitted trajectories. +Assignment 4: Return to the Pixel Time-Series Options section of the control panel, and navigate to a pixel in your area of interest that you believe would show a distinctive recovery or growth process, as you define it for your topic of interest. You may want to modify the index, parameters, etc. as covered in Section 1 to adequately capture the growth process with the fitted trajectories. -* Question 7. Did you use the same spectral index? If not, why? -* Question 8. Were the fitting parameters the same as those for disturbance? If not, what did you change, and why? -* Question 9. What evidence do you have that this is a vegetative growth signal? +* Question 7. Did you use the same spectral index? If not, why? +* Question 8. Were the fitting parameters the same as those for disturbance? If not, what did you change, and why? +* Question 9. What evidence do you have that this is a vegetative growth signal? -Assignment 5. For vegetation gain mapping, switch the control panel back to Change Filter Options and use the guidance in Section 2 to set parameters, etc. to make maps of growth. +Assignment 5. For vegetation gain mapping, switch the control panel back to Change Filter Options and use the guidance in Section 2 to set parameters, etc. to make maps of growth. -* Question 10. For the pixel or pixels you found for Assignment 3, does the year and magnitude as mapped in the “gain” image match with what you would expect from the trajectory itself? -* Question 11. Compare what the map looks like when you run it with and without the MMU filter. What differences do you see? -* Question 12. Try changing the recovery duration filter to a very high number (perhaps the full length of your archive) and to a very low number (say, one or two years). What differences do you see?  +* Question 10. For the pixel or pixels you found for Assignment 3, does the year and magnitude as mapped in the “gain” image match with what you would expect from the trajectory itself? +* Question 11. Compare what the map looks like when you run it with and without the MMU filter. What differences do you see? +* Question 12. Try changing the recovery duration filter to a very high number (perhaps the full length of your archive) and to a very low number (say, one or two years). What differences do you see? ## Conclusion {.unnumbered} This exercise provides a baseline sense of how the LandTrendr algorithm works. The key points are learning how to interpret change in spectral values in terms of the processes occurring on the ground, and then translating those into maps. -You can export the images you’ve made here using Download Options. Links to materials are available in the chapter checkpoints and LandTrendr documentation about both the GUI and the script-based versions of the algorithm. In particular, there are scripts that handle different components of the fitting and mapping process, and that allow you to keep track of the fitting and image selection criteria. +You can export the images you’ve made here using Download Options. Links to materials are available in the chapter checkpoints and LandTrendr documentation about both the GUI and the script-based versions of the algorithm. In particular, there are scripts that handle different components of the fitting and mapping process, and that allow you to keep track of the fitting and image selection criteria. ## References {.unnumbered} @@ -1535,7 +1555,7 @@ Kennedy RE, Yang Z, Gorelick N, et al (2018) Implementation of the LandTrendr al -::: {.callout-tip} +:::{.callout-tip} # Chapter Information ## Author {.unlisted .unnumbered} @@ -1549,7 +1569,7 @@ Andréa Puzzi Nicolau, Karen Dyson, Biplov Bhandari, David Saah, Nicholas Clinto ## Overview {.unlisted .unnumbered} -The purpose of this chapter is to establish a foundation for time-series analysis of remotely sensed data, which is typically arranged as an ordered stack of images. You will be introduced to the concepts of graphing time series, using linear modeling to detrend time series, and fitting harmonic models to time-series data. At the completion of this chapter, you will be able to perform analysis of multi-temporal data for determining trend and seasonality on a per-pixel basis. +The purpose of this chapter is to establish a foundation for time-series analysis of remotely sensed data, which is typically arranged as an ordered stack of images. You will be introduced to the concepts of graphing time series, using linear modeling to detrend time series, and fitting harmonic models to time-series data. At the completion of this chapter, you will be able to perform analysis of multi-temporal data for determining trend and seasonality on a per-pixel basis. ## Learning Outcomes {.unlisted .unnumbered} @@ -1563,348 +1583,354 @@ The purpose of this chapter is to establish a foundation for time-series analysi * Import images and image collections, filter, and visualize (Part F1). * Perform basic image analysis: select bands, compute indices, create masks (Part F2). -* ​​Create a graph using ui.Chart (Chap. F1.3). -* Use normalizedDifference to calculate vegetation indices (Chap. F2.0). -* Write a function and map it over an ImageCollection (Chap. F4.0). +* ​​Create a graph using ui.Chart (Chap. F1.3). +* Use normalizedDifference to calculate vegetation indices (Chap. F2.0). +* Write a function and map it over an ImageCollection (Chap. F4.0). * Mask cloud, cloud shadow, snow/ice, and other undesired pixels (Chap. F4.3). -::: +::: ## Introduction {.unlisted .unnumbered} -Many natural and man-made phenomena exhibit important annual, interannual, or longer-term trends that recur—that is, they occur at roughly regular intervals. Examples include seasonality in leaf patterns in deciduous forests and seasonal crop growth patterns. Over time, indices such as the Normalized Difference Vegetation Index (NDVI) will show regular increases (e.g., leaf-on, crop growth) and decreases (e.g., leaf-off, crop senescence), and typically have a long-term, if noisy, trend such as a gradual increase in NDVI value as an area recovers from a disturbance. +Many natural and man-made phenomena exhibit important annual, interannual, or longer-term trends that recur—that is, they occur at roughly regular intervals. Examples include seasonality in leaf patterns in deciduous forests and seasonal crop growth patterns. Over time, indices such as the Normalized Difference Vegetation Index (NDVI) will show regular increases (e.g., leaf-on, crop growth) and decreases (e.g., leaf-off, crop senescence), and typically have a long-term, if noisy, trend such as a gradual increase in NDVI value as an area recovers from a disturbance. -Earth Engine supports the ability to do complex linear and non-linear regressions of values in each pixel of a study area. Simple linear regressions of indices can reveal linear trends that can span multiple years. Meanwhile, harmonic terms can be used to fit a sine-wave-like curve. Once you have the ability to fit these functions to time series, you can answer many important questions. For example, you can define vegetation dynamics over multiple time scales, identify phenology and track changes year to year, and identify deviations from the expected patterns (Bradley et al. 2007, Bullock et al. 2020). There are multiple applications for these analyses. For example, algorithms to detect deviations from the expected pattern can be used to identify disturbance events, including deforestation and forest degradation (Bullock et al. 2020). +Earth Engine supports the ability to do complex linear and non-linear regressions of values in each pixel of a study area. Simple linear regressions of indices can reveal linear trends that can span multiple years. Meanwhile, harmonic terms can be used to fit a sine-wave-like curve. Once you have the ability to fit these functions to time series, you can answer many important questions. For example, you can define vegetation dynamics over multiple time scales, identify phenology and track changes year to year, and identify deviations from the expected patterns (Bradley et al. 2007, Bullock et al. 2020). There are multiple applications for these analyses. For example, algorithms to detect deviations from the expected pattern can be used to identify disturbance events, including deforestation and forest degradation (Bullock et al. 2020). -If you have not already done so, be sure to add the book’s code repository to the Code Editor by entering [https://code.earthengine.google.com/?accept_repo=projects/gee-edu/book](https://www.google.com/url?q=https://code.earthengine.google.com/?accept_repo%3Dprojects/gee-edu/book&sa=D&source=editors&ust=1671458868327546&usg=AOvVaw0m0oT1feMKQKRq3rtuOxfY) into your browser. The book’s scripts will then be available in the script manager panel. +If you have not already done so, be sure to add the book’s code repository to the Code Editor by entering [https://code.earthengine.google.com/?accept_repo=projects/gee-edu/book](https://www.google.com/url?q=https://code.earthengine.google.com/?accept_repo%3Dprojects/gee-edu/book&sa=D&source=editors&ust=1671458868327546&usg=AOvVaw0m0oT1feMKQKRq3rtuOxfY) into your browser. The book’s scripts will then be available in the script manager panel. ## Multi-Temporal Data in Earth Engine -As explained in Chaps. F4.0 and F4.1, a time series in Earth Engine is typically represented as an ImageCollection. Because of image overlaps, cloud treatments, and filtering choices, an ImageCollection can have any of the following complex characteristics: +As explained in Chaps. F4.0 and F4.1, a time series in Earth Engine is typically represented as an ImageCollection. Because of image overlaps, cloud treatments, and filtering choices, an ImageCollection can have any of the following complex characteristics: * At each pixel, there might be a distinct number of observations taken from a unique set of dates. * The size (length) of the time series can vary across pixels. * Data may be missing in any pixel at any point in the sequence (e.g., due to cloud masking). -The use of multi-temporal data in Earth Engine introduces two mind-bending concepts, which we will describe below. +The use of multi-temporal data in Earth Engine introduces two mind-bending concepts, which we will describe below. -Per-pixel curve fitting. As you have likely encountered in many settings, a function can be fit through a series of values. In the most familiar example, a function of the form y = mx + b can represent a linear trend in data of all kinds. Fitting a straight “curve” with linear regression techniques involves estimating m and b for a set of x and y values. In a time series, x typically represents time, while y values represent observations at specific times. This chapter introduces how to estimate m and b for computed indices through time to model a potential linear trend in a time series. We then demonstrate how to fit a sinusoidal wave, which is useful for modeling rising and falling values, such as NDVI over a growing season. What can be particularly mind-bending in this setting is the fact that when Earth Engine is asked to estimate values across a large area, it will fit a function in every pixel of the study area. Each pixel, then, has its own m and b values, determined by the number of observations in that pixel, the observed values, and the dates for which they were observed. +Per-pixel curve fitting. As you have likely encountered in many settings, a function can be fit through a series of values. In the most familiar example, a function of the form y = mx + b can represent a linear trend in data of all kinds. Fitting a straight “curve” with linear regression techniques involves estimating m and b for a set of x and y values. In a time series, x typically represents time, while y values represent observations at specific times. This chapter introduces how to estimate m and b for computed indices through time to model a potential linear trend in a time series. We then demonstrate how to fit a sinusoidal wave, which is useful for modeling rising and falling values, such as NDVI over a growing season. What can be particularly mind-bending in this setting is the fact that when Earth Engine is asked to estimate values across a large area, it will fit a function in every pixel of the study area. Each pixel, then, has its own m and b values, determined by the number of observations in that pixel, the observed values, and the dates for which they were observed. -Higher-dimension band values: array images. That more complex conception of the potential information contained in a single pixel can be represented in a higher-order Earth Engine structure: the array image. As you will encounter in this lab, it is possible for a single pixel in a single band of a single image to contain more than one value. If you choose to implement an array image, a single pixel might contain a one-dimensional vector of numbers, perhaps holding the slope and intercept values resulting from a linear regression, for example. Other examples, outside the scope of this chapter but used in the next chapter, might employ a two-dimensional matrix of values for each pixel within a single band of an image. Higher-order dimensions are available, as well as array image manipulations borrowed from the world of matrix algebra. Additionally, there are functions to move between the multidimensional array image structure and the more familiar, more easily displayed, simple Image type. Some of these array image functions were encountered in Chap. F3.1, but with less explanatory context. +Higher-dimension band values: array images. That more complex conception of the potential information contained in a single pixel can be represented in a higher-order Earth Engine structure: the array image. As you will encounter in this lab, it is possible for a single pixel in a single band of a single image to contain more than one value. If you choose to implement an array image, a single pixel might contain a one-dimensional vector of numbers, perhaps holding the slope and intercept values resulting from a linear regression, for example. Other examples, outside the scope of this chapter but used in the next chapter, might employ a two-dimensional matrix of values for each pixel within a single band of an image. Higher-order dimensions are available, as well as array image manipulations borrowed from the world of matrix algebra. Additionally, there are functions to move between the multidimensional array image structure and the more familiar, more easily displayed, simple Image type. Some of these array image functions were encountered in Chap. F3.1, but with less explanatory context. -First, we will give some very basic notation (Fig. F4.6.1). A scalar pixel at time t is given by pt, and a pixel vector by pt. A variable with a “hat” represents an estimated value: in this context, p̂t is the estimated pixel value at time t. A time series is a collection of pixel values, usually sorted chronologically: {pt; t = t0...tN}, where t might be in any units, t0 is the smallest, and tN is the largest such t in the series. +First, we will give some very basic notation (Fig. F4.6.1). A scalar pixel at time t is given by pt, and a pixel vector by pt. A variable with a “hat” represents an estimated value: in this context, p̂t is the estimated pixel value at time t. A time series is a collection of pixel values, usually sorted chronologically: {pt; t = t0...tN}, where t might be in any units, t0 is the smallest, and tN is the largest such t in the series. -![](F4/image38.png) +![Fig. F4.6.1 Time series representation of pixel p ](F4/image38.png) -Fig. F4.6.1 Time series representation of pixel p  ## Data Preparation and Preprocessing -The first step in analysis of time-series data is to import data of interest and plot it at an interesting location. We will work with the USGS Landsat 8 Level 2, Collection 2, Tier 1 ImageCollection and a cloud-masking function (Chap. F4.3), scale the image values, and add variables of interest to the collection as bands. Copy and paste the code below to filter the Landsat 8 collection to a point of interest over California (variable roi) and specific dates, and to apply the defined function. The variables of interest added by the function are: (1) NDVI (Chap. F2.0), (2) a time variable that is the difference between the image’s current year and the year 1970 (a start point), and (3) a constant variable with value 1. +The first step in analysis of time-series data is to import data of interest and plot it at an interesting location. We will work with the USGS Landsat 8 Level 2, Collection 2, Tier 1 ImageCollection and a cloud-masking function (Chap. F4.3), scale the image values, and add variables of interest to the collection as bands. Copy and paste the code below to filter the Landsat 8 collection to a point of interest over California (variable roi) and specific dates, and to apply the defined function. The variables of interest added by the function are: (1) NDVI (Chap. F2.0), (2) a time variable that is the difference between the image’s current year and the year 1970 (a start point), and (3) a constant variable with value 1. +```js ///////////////////// Sections 1 & 2 ///////////////////////////// // Define function to mask clouds, scale, and add variables // (NDVI, time and a constant) to Landsat 8 imagery. -function maskScaleAndAddVariable(image) {   // Bit 0 - Fill   // Bit 1 - Dilated Cloud   // Bit 2 - Cirrus   // Bit 3 - Cloud   // Bit 4 - Cloud Shadow   var qaMask = image.select('QA_PIXEL').bitwiseAnd(parseInt('11111',       2)).eq(0);   var saturationMask = image.select('QA_RADSAT').eq(0);   // Apply the scaling factors to the appropriate bands.   var opticalBands = image.select('SR_B.').multiply(0.0000275).add(-       0.2);   var thermalBands = image.select('ST_B.*').multiply(0.00341802) -       .add(149.0);   // Replace the original bands with the scaled ones and apply the masks.   var img = image.addBands(opticalBands, null, true) -       .addBands(thermalBands, null, true) -       .updateMask(qaMask) -       .updateMask(saturationMask);   var imgScaled = image.addBands(img, null, true);   // Now we start to add variables of interest.   // Compute time in fractional years since the epoch.   var date = ee.Date(image.get('system:time_start'));   var years = date.difference(ee.Date('1970-01-01'), 'year');   // Return the image with the added bands.   return imgScaled       // Add an NDVI band.       .addBands(imgScaled.normalizedDifference(['SR_B5', 'SR_B4']) -           .rename('NDVI'))       // Add a time band.       .addBands(ee.Image(years).rename('t')) -       .float()       // Add a constant band.       .addBands(ee.Image.constant(1)); +function maskScaleAndAddVariable(image) { // Bit 0 - Fill // Bit 1 - Dilated Cloud // Bit 2 - Cirrus // Bit 3 - Cloud // Bit 4 - Cloud Shadow var qaMask = image.select('QA_PIXEL').bitwiseAnd(parseInt('11111', 2)).eq(0); var saturationMask = image.select('QA_RADSAT').eq(0); // Apply the scaling factors to the appropriate bands. var opticalBands = image.select('SR_B.').multiply(0.0000275).add(- 0.2); var thermalBands = image.select('ST_B.*').multiply(0.00341802) + .add(149.0); // Replace the original bands with the scaled ones and apply the masks. var img = image.addBands(opticalBands, null, true) + .addBands(thermalBands, null, true) + .updateMask(qaMask) + .updateMask(saturationMask); var imgScaled = image.addBands(img, null, true); // Now we start to add variables of interest. // Compute time in fractional years since the epoch. var date = ee.Date(image.get('system:time_start')); var years = date.difference(ee.Date('1970-01-01'), 'year'); // Return the image with the added bands. return imgScaled // Add an NDVI band. .addBands(imgScaled.normalizedDifference(['SR_B5', 'SR_B4']) + .rename('NDVI')) // Add a time band. .addBands(ee.Image(years).rename('t')) + .float() // Add a constant band. .addBands(ee.Image.constant(1)); } // Import point of interest over California, USA. -var roi = ee.Geometry.Point([-121.059, 37.9242]); +var roi = ee.Geometry.Point([-121.059, 37.9242]); // Import the USGS Landsat 8 Level 2, Collection 2, Tier 1 image collection), // filter, mask clouds, scale, and add variables. -var landsat8sr = ee.ImageCollection('LANDSAT/LC08/C02/T1_L2') -   .filterBounds(roi) -   .filterDate('2013-01-01', '2018-01-01') -   .map(maskScaleAndAddVariable); +var landsat8sr = ee.ImageCollection('LANDSAT/LC08/C02/T1_L2') + .filterBounds(roi) + .filterDate('2013-01-01', '2018-01-01') + .map(maskScaleAndAddVariable); // Set map center over the ROI. Map.centerObject(roi, 6); -Next, to visualize the NDVI at the point of interest over time, copy and paste the code below to print a chart of the time series (Chap. F1.3) at the location of interest (Fig. F4.6.2). +``` +Next, to visualize the NDVI at the point of interest over time, copy and paste the code below to print a chart of the time series (Chap. F1.3) at the location of interest (Fig. F4.6.2). +```js // Plot a time series of NDVI at a single location. -var landsat8Chart = ui.Chart.image.series(landsat8sr.select('NDVI'), roi) -   .setChartType('ScatterChart') -   .setOptions({ -       title: 'Landsat 8 NDVI time series at ROI', -       lineWidth: 1, -       pointSize: 3, -   }); +var landsat8Chart = ui.Chart.image.series(landsat8sr.select('NDVI'), roi) + .setChartType('ScatterChart') + .setOptions({ + title: 'Landsat 8 NDVI time series at ROI', + lineWidth: 1, + pointSize: 3, + }); print(landsat8Chart); -![](F4/image3.png) +``` +![Fig. F4.6.2 Time series representation of pixel p ](F4/image3.png) -Fig. F4.6.2 Time series representation of pixel p  -We can add a linear trend line to our chart using the trendlines parameters in the setOptions function for image series charts. Copy and paste the code below to print the same chart but with a linear trend line plotted (Fig. F4.6.3). In the next section, you will learn how to estimate linear trends over time. +We can add a linear trend line to our chart using the trendlines parameters in the setOptions function for image series charts. Copy and paste the code below to print the same chart but with a linear trend line plotted (Fig. F4.6.3). In the next section, you will learn how to estimate linear trends over time. +```js // Plot a time series of NDVI with a linear trend line // at a single location. -var landsat8ChartTL = ui.Chart.image.series(landsat8sr.select('NDVI'), roi) -   .setChartType('ScatterChart') -   .setOptions({ -       title: 'Landsat 8 NDVI time series at ROI', -       trendlines: {           0: { -               color: 'CC0000'           } -       }, -       lineWidth: 1, -       pointSize: 3, -   }); +var landsat8ChartTL = ui.Chart.image.series(landsat8sr.select('NDVI'), roi) + .setChartType('ScatterChart') + .setOptions({ + title: 'Landsat 8 NDVI time series at ROI', + trendlines: { 0: { + color: 'CC0000' } + }, + lineWidth: 1, + pointSize: 3, + }); print(landsat8ChartTL); -![](F4/image82.png) +``` +![Fig. F4.6.3 Time series representation of pixel p with the trend line in red](F4/image82.png) -Fig. F4.6.3 Time series representation of pixel p with the trend line in red Now that we have plotted and visualized the data, lots of interesting analyses can be done to the time series by harnessing Earth Engine tools for fitting curves through this data. We will see a couple of examples in the following sections. -::: {.callout-note} -Code Checkpoint F46a. The book’s repository contains a script that shows what your code should look like at this point. +:::{.callout-note} +Code Checkpoint F46a. The book’s repository contains a script that shows what your code should look like at this point. ::: -## Estimating Linear Trend Over Time +## Estimating Linear Trend Over Time Time series datasets may contain not only trends but also seasonality, both of which may need to be removed prior to modeling. Trends and seasonality can result in a varying mean and a varying variance over time, both of which define a time series as non-stationary. Stationary datasets, on the other hand, have a stable mean and variance, and are therefore much easier to model. -Consider the following linear model, where et is a random error: +Consider the following linear model, where et is a random error: -pt = β0 + β1t + et                                                        (Eq. F4.6.1) +pt = β0 + β1t + et (Eq. F4.6.1) This is the model behind the trend line added to the chart created in the previous section (Fig. F4.6.3). Identifying trends at different scales is a big topic, with many approaches being used (e.g., differencing, modeling). -Removing unwanted to uninteresting trends for a given problem is often a first step to understanding complex patterns in time series. There are several approaches to remove trends. Here, we will remove the linear trend that is evident in the data shown in Fig. F4.6.3 using Earth Engine’s built-in tools for regression modeling. This approach is a useful, straightforward way to detrend data in time series (Shumway and Stoffer 2019). Here, the goal is to discover the values of the β’s in Eq. F4.6.1 for each pixel. +Removing unwanted to uninteresting trends for a given problem is often a first step to understanding complex patterns in time series. There are several approaches to remove trends. Here, we will remove the linear trend that is evident in the data shown in Fig. F4.6.3 using Earth Engine’s built-in tools for regression modeling. This approach is a useful, straightforward way to detrend data in time series (Shumway and Stoffer 2019). Here, the goal is to discover the values of the β’s in Eq. F4.6.1 for each pixel. -Copy and paste code below into the Code Editor, adding it to the end of the script from the previous section. Running this code will fit this trend model to the Landsat-based NDVI series using ordinary least squares, using the linearRegression reducer (Chap. F3.0). +Copy and paste code below into the Code Editor, adding it to the end of the script from the previous section. Running this code will fit this trend model to the Landsat-based NDVI series using ordinary least squares, using the linearRegression reducer (Chap. F3.0). +```js ///////////////////// Section 3 ///////////////////////////// // List of the independent variable names -var independents = ee.List(['constant', 't']); +var independents = ee.List(['constant', 't']); // Name of the dependent variable. -var dependent = ee.String('NDVI'); +var dependent = ee.String('NDVI'); -// Compute a linear trend.  This will have two bands: 'residuals' and +// Compute a linear trend. This will have two bands: 'residuals' and // a 2x1 (Array Image) band called 'coefficients'. // (Columns are for dependent variables) -var trend = landsat8sr.select(independents.add(dependent)) -   .reduce(ee.Reducer.linearRegression(independents.length(), 1)); +var trend = landsat8sr.select(independents.add(dependent)) + .reduce(ee.Reducer.linearRegression(independents.length(), 1)); Map.addLayer(trend, {}, 'trend array image'); // Flatten the coefficients into a 2-band image. -var coefficients = trend.select('coefficients')   // Get rid of extra dimensions and convert back to a regular image   .arrayProject([0]) -   .arrayFlatten([independents]); +var coefficients = trend.select('coefficients') // Get rid of extra dimensions and convert back to a regular image .arrayProject([0]) + .arrayFlatten([independents]); Map.addLayer(coefficients, {}, 'coefficients image'); -If you click over a point using the Inspector tab, you will see the pixel values for the array image (coefficients “t” and “constant”, and residuals) and two-band image (coefficients “t” and “constant”) (Fig. F4.6.4). +``` +If you click over a point using the Inspector tab, you will see the pixel values for the array image (coefficients “t” and “constant”, and residuals) and two-band image (coefficients “t” and “constant”) (Fig. F4.6.4). -![](F4/image53.png) +![Fig. F4.6.4 Pixel values of array image and coefficients image](F4/image53.png) -Fig. F4.6.4 Pixel values of array image and coefficients image -Now, copy and paste the code below to use the model to detrend the original NDVI time series and plot the time series chart with the trendlines parameter (Fig. F4.6.5). +Now, copy and paste the code below to use the model to detrend the original NDVI time series and plot the time series chart with the trendlines parameter (Fig. F4.6.5). -// Compute a detrended series. -var detrended = landsat8sr.map(function(image) {   return image.select(dependent).subtract( -           image.select(independents).multiply(coefficients) -           .reduce('sum')) -       .rename(dependent) -       .copyProperties(image, ['system:time_start']); +```js +// Compute a detrended series. +var detrended = landsat8sr.map(function(image) { return image.select(dependent).subtract( + image.select(independents).multiply(coefficients) + .reduce('sum')) + .rename(dependent) + .copyProperties(image, ['system:time_start']); }); // Plot the detrended results. -var detrendedChart = ui.Chart.image.series(detrended, roi, null, 30) -   .setOptions({ -       title: 'Detrended Landsat time series at ROI', -       lineWidth: 1, -       pointSize: 3, -       trendlines: {           0: { -               color: 'CC0000'           } -       }, -   });print(detrendedChart); +var detrendedChart = ui.Chart.image.series(detrended, roi, null, 30) + .setOptions({ + title: 'Detrended Landsat time series at ROI', + lineWidth: 1, + pointSize: 3, + trendlines: { 0: { + color: 'CC0000' } + }, + });print(detrendedChart); -![](F4/image20.png) +``` +![Fig. F4.6.5 Detrended NDVI time series](F4/image20.png) -Fig. F4.6.5 Detrended NDVI time series -::: {.callout-note} -Code Checkpoint F46b. The book’s repository contains a script that shows what your code should look like at this point. +:::{.callout-note} +Code Checkpoint F46b. The book’s repository contains a script that shows what your code should look like at this point. ::: ## Estimating Seasonality with a Harmonic Model -A linear trend is one of several possible types of trends in time series. Time series can also present harmonic trends, in which a value goes up and down in a predictable wave pattern. These are of particular interest and usefulness in the natural world, where harmonic changes in greenness of deciduous vegetation can occur across the spring, summer, and autumn. Now we will return to the initial time series (landsat8sr) of Fig. F4.6.2 and fit a harmonic pattern through the data. Consider the following harmonic model, where A is amplitude, ω is frequency, φ is phase, and et is a random error. +A linear trend is one of several possible types of trends in time series. Time series can also present harmonic trends, in which a value goes up and down in a predictable wave pattern. These are of particular interest and usefulness in the natural world, where harmonic changes in greenness of deciduous vegetation can occur across the spring, summer, and autumn. Now we will return to the initial time series (landsat8sr) of Fig. F4.6.2 and fit a harmonic pattern through the data. Consider the following harmonic model, where A is amplitude, ω is frequency, φ is phase, and et is a random error. -pt = β0 + β1t + Acos(2πωt - φ) + et  +pt = β0 + β1t + Acos(2πωt - φ) + et -    = β0 + β1t + β2cos(2πωt) + β3sin(2πωt) + et                (Eq. F4.6.2) + = β0 + β1t + β2cos(2πωt) + β3sin(2πωt) + et (Eq. F4.6.2) -Note that β2 = Acos(φ) and β3 = Asin(φ), implying A = (β22 + β32)½ and φ = atan(β3/β2) (as described in Shumway and Stoffer 2019). To fit this model to an annual time series, set ω = 1 (one cycle per year) and use ordinary least squares regression. +Note that β2 = Acos(φ) and β3 = Asin(φ), implying A = (β22 + β32)½ and φ = atan(β3/β2) (as described in Shumway and Stoffer 2019). To fit this model to an annual time series, set ω = 1 (one cycle per year) and use ordinary least squares regression. -The setup for fitting the model is to first add the harmonic variables (the third and fourth terms of Eq. F4.6.2) to the ImageCollection. Then, fit the model as with the linear trend, using the linearRegression reducer, which will yield a 4 x 1 array image. +The setup for fitting the model is to first add the harmonic variables (the third and fourth terms of Eq. F4.6.2) to the ImageCollection. Then, fit the model as with the linear trend, using the linearRegression reducer, which will yield a 4 x 1 array image. +```js ///////////////////// Section 4 ///////////////////////////// // Use these independent variables in the harmonic regression. -var harmonicIndependents = ee.List(['constant', 't', 'cos', 'sin']); +var harmonicIndependents = ee.List(['constant', 't', 'cos', 'sin']); // Add harmonic terms as new image bands. -var harmonicLandsat = landsat8sr.map(function(image) {   var timeRadians = image.select('t').multiply(2 * Math.PI);   return image       .addBands(timeRadians.cos().rename('cos')) -       .addBands(timeRadians.sin().rename('sin')); +var harmonicLandsat = landsat8sr.map(function(image) { var timeRadians = image.select('t').multiply(2 * Math.PI); return image .addBands(timeRadians.cos().rename('cos')) + .addBands(timeRadians.sin().rename('sin')); }); // Fit the model. -var harmonicTrend = harmonicLandsat -   .select(harmonicIndependents.add(dependent))   // The output of this reducer is a 4x1 array image.   .reduce(ee.Reducer.linearRegression(harmonicIndependents.length(),       1)); +var harmonicTrend = harmonicLandsat + .select(harmonicIndependents.add(dependent)) // The output of this reducer is a 4x1 array image. .reduce(ee.Reducer.linearRegression(harmonicIndependents.length(), 1)); -Now, copy and paste the code below to plug the coefficients into Eq. F4.6.2 in order to get a time series of fitted values and plot the harmonic model time series (Fig. F4.6.6). +``` +Now, copy and paste the code below to plug the coefficients into Eq. F4.6.2 in order to get a time series of fitted values and plot the harmonic model time series (Fig. F4.6.6). +```js // Turn the array image into a multi-band image of coefficients. -var harmonicTrendCoefficients = harmonicTrend.select('coefficients') -   .arrayProject([0]) -   .arrayFlatten([harmonicIndependents]); +var harmonicTrendCoefficients = harmonicTrend.select('coefficients') + .arrayProject([0]) + .arrayFlatten([harmonicIndependents]); // Compute fitted values. -var fittedHarmonic = harmonicLandsat.map(function(image) {   return image.addBands( -       image.select(harmonicIndependents) -       .multiply(harmonicTrendCoefficients) -       .reduce('sum') -       .rename('fitted')); +var fittedHarmonic = harmonicLandsat.map(function(image) { return image.addBands( + image.select(harmonicIndependents) + .multiply(harmonicTrendCoefficients) + .reduce('sum') + .rename('fitted')); }); // Plot the fitted model and the original data at the ROI. print(ui.Chart.image.series( -       fittedHarmonic.select(['fitted', 'NDVI']), roi, ee.Reducer -       .mean(), 30) -   .setSeriesNames(['NDVI', 'fitted']) -   .setOptions({ -       title: 'Harmonic model: original and fitted values', -       lineWidth: 1, -       pointSize: 3, -   })); + fittedHarmonic.select(['fitted', 'NDVI']), roi, ee.Reducer + .mean(), 30) + .setSeriesNames(['NDVI', 'fitted']) + .setOptions({ + title: 'Harmonic model: original and fitted values', + lineWidth: 1, + pointSize: 3, + })); -![](F4/image86.png) +``` +![Fig. F4.6.6 Harmonic model of NDVI time series](F4/image86.png) -Fig. F4.6.6 Harmonic model of NDVI time series -Returning to the mind-bending nature of curve-fitting, it is worth remembering that the harmonic waves seen in Fig. F4.6.6 are the fit of the data to a single point across the image. Next, we will map the outcomes of millions of these fits, pixel by pixel, across the entire study area. +Returning to the mind-bending nature of curve-fitting, it is worth remembering that the harmonic waves seen in Fig. F4.6.6 are the fit of the data to a single point across the image. Next, we will map the outcomes of millions of these fits, pixel by pixel, across the entire study area. We’ll compute and map the phase and amplitude of the estimated harmonic model for each pixel. Phase and amplitude (Fig. F4.6.7) can give us additional information to facilitate remote sensing applications such as agricultural mapping and land use and land cover monitoring. Agricultural crops with different phenological cycles can be distinguished with phase and amplitude information, something that perhaps would not be possible with spectral information alone. -![](F4/image8.png) +![Fig. F4.6.7 Example of phase and amplitude in harmonic model](F4/image8.png) -Fig. F4.6.7 Example of phase and amplitude in harmonic model Copy and paste the code below to compute phase and amplitude from the coefficients and add this image to the map (Fig. F4.6.8). +```js // Compute phase and amplitude. -var phase = harmonicTrendCoefficients.select('sin') -   .atan2(harmonicTrendCoefficients.select('cos'))   // Scale to [0, 1] from radians.   .unitScale(-Math.PI, Math.PI); +var phase = harmonicTrendCoefficients.select('sin') + .atan2(harmonicTrendCoefficients.select('cos')) // Scale to [0, 1] from radians. .unitScale(-Math.PI, Math.PI); -var amplitude = harmonicTrendCoefficients.select('sin') -   .hypot(harmonicTrendCoefficients.select('cos'))   // Add a scale factor for visualization.   .multiply(5); +var amplitude = harmonicTrendCoefficients.select('sin') + .hypot(harmonicTrendCoefficients.select('cos')) // Add a scale factor for visualization. .multiply(5); // Compute the mean NDVI. -var meanNdvi = landsat8sr.select('NDVI').mean(); +var meanNdvi = landsat8sr.select('NDVI').mean(); // Use the HSV to RGB transformation to display phase and amplitude. -var rgb = ee.Image.cat([ -   phase, // hue   amplitude, // saturation (difference from white)   meanNdvi // value (difference from black) +var rgb = ee.Image.cat([ + phase, // hue amplitude, // saturation (difference from white) meanNdvi // value (difference from black) ]).hsvToRgb(); Map.addLayer(rgb, {}, 'phase (hue), amplitude (sat), ndvi (val)'); -![](F4/image63.png) +``` +![Fig. F4.6.8 Phase, amplitude, and NDVI concatenated image](F4/image63.png) -Fig. F4.6.8 Phase, amplitude, and NDVI concatenated image -The code uses the HSV to RGB transformation hsvToRgb for visualization purposes (Chap. F3.1). We use this transformation to separate color components from intensity for a better visualization. Without this transformation, we would visualize a very colorful image that would not look as intuitive as the image with the transformation. With this transformation, phase, amplitude, and mean NDVI are displayed in terms of hue (color), saturation (difference from white), and value (difference from black), respectively. Therefore, darker pixels are areas with low NDVI. For example, water bodies will appear as black, since NDVI values are zero or negative. The different colors are distinct phase values, and the saturation of the color refers to the amplitude: whiter colors mean amplitude closer to zero (e.g., forested areas), and the more vivid the colors, the higher the amplitude (e.g., croplands). Note that if you use the Inspector tool to analyze the values of a pixel, you will not get values of phase, amplitude, and NDVI, but the transformed values into values of blue, green, and red colors. +The code uses the HSV to RGB transformation hsvToRgb for visualization purposes (Chap. F3.1). We use this transformation to separate color components from intensity for a better visualization. Without this transformation, we would visualize a very colorful image that would not look as intuitive as the image with the transformation. With this transformation, phase, amplitude, and mean NDVI are displayed in terms of hue (color), saturation (difference from white), and value (difference from black), respectively. Therefore, darker pixels are areas with low NDVI. For example, water bodies will appear as black, since NDVI values are zero or negative. The different colors are distinct phase values, and the saturation of the color refers to the amplitude: whiter colors mean amplitude closer to zero (e.g., forested areas), and the more vivid the colors, the higher the amplitude (e.g., croplands). Note that if you use the Inspector tool to analyze the values of a pixel, you will not get values of phase, amplitude, and NDVI, but the transformed values into values of blue, green, and red colors. -::: {.callout-note} -Code Checkpoint F46c. The book’s repository contains a script that shows what your code should look like at this point. +:::{.callout-note} +Code Checkpoint F46c. The book’s repository contains a script that shows what your code should look like at this point. ::: ## An Application of Curve Fitting ### The rich data about the curve fits can be viewed in a multitude of different ways. Add the code below to your script to produce the view in Fig. 4.6.9. The image will be a close-up of the area around Modesto, California. +```js ///////////////////// Section 5 ///////////////////////////// // Import point of interest over California, USA. -var roi = ee.Geometry.Point([-121.04, 37.641]); +var roi = ee.Geometry.Point([-121.04, 37.641]); // Set map center over the ROI. Map.centerObject(roi, 14); -var trend0D = trend.select('coefficients').arrayProject([0]) -   .arrayFlatten([independents]).select('t'); +var trend0D = trend.select('coefficients').arrayProject([0]) + .arrayFlatten([independents]).select('t'); -var anotherView = ee.Image(harmonicTrendCoefficients.select('sin')) -   .addBands(trend0D) -   .addBands(harmonicTrendCoefficients.select('cos')); +var anotherView = ee.Image(harmonicTrendCoefficients.select('sin')) + .addBands(trend0D) + .addBands(harmonicTrendCoefficients.select('cos')); Map.addLayer(anotherView, -   { -       min: -0.03, -       max: 0.03   },   'Another combination of fit characteristics'); + { + min: -0.03, + max: 0.03 }, 'Another combination of fit characteristics'); +``` ![](F4/image61.png) -![](F4/image45.png) +![Fig. F4.6.9 Two views of the harmonic fits for NDVI for the Modesto, California area](F4/image45.png) -Fig. F4.6.9 Two views of the harmonic fits for NDVI for the Modesto, California area -The upper image in Fig. F4.6.9 is a closer view of Fig. F4.6.8, showing an image that transforms the sine and cosine coefficient values, and incorporates information from the mean NDVI. The lower image draws the sine and cosine in the red and blue bands, and extracts the slope of the linear trend that you calculated earlier in the chapter, placing that in the green band. The two views of the fit are similarly structured in their spatial pattern—both show fields to the west and the city to the east. But the pixel-by-pixel variability emphasizes a key point of this chapter: that a fit to the NDVI data is done independently in each pixel in the image. Using different elements of the fit, these two views, like other combinations of the data you might imagine, can reveal the rich variability of the landscape around Modesto. +The upper image in Fig. F4.6.9 is a closer view of Fig. F4.6.8, showing an image that transforms the sine and cosine coefficient values, and incorporates information from the mean NDVI. The lower image draws the sine and cosine in the red and blue bands, and extracts the slope of the linear trend that you calculated earlier in the chapter, placing that in the green band. The two views of the fit are similarly structured in their spatial pattern—both show fields to the west and the city to the east. But the pixel-by-pixel variability emphasizes a key point of this chapter: that a fit to the NDVI data is done independently in each pixel in the image. Using different elements of the fit, these two views, like other combinations of the data you might imagine, can reveal the rich variability of the landscape around Modesto. -::: {.callout-note} -Code Checkpoint F46d. The book’s repository contains a script that shows what your code should look like at this point. +:::{.callout-note} +Code Checkpoint F46d. The book’s repository contains a script that shows what your code should look like at this point. ::: -## Higher-Order Harmonic Models +## Higher-Order Harmonic Models Harmonic models are not limited to fitting a single wave through a set of points. In some situations, there may be more than one cycle within a given year—for example, when an agricultural field is double-cropped. Modeling multiple waves within a given year can be done by adding more harmonic terms to Eq. F4.6.2. The code at the following checkpoint allows the fitting of any number of cycles through a given point. -::: {.callout-note} -Code Checkpoint F46e. The book’s repository contains a script to use to begin this section. You will need to start with that script and edit the code to produce the charts in this section. +:::{.callout-note} +Code Checkpoint F46e. The book’s repository contains a script to use to begin this section. You will need to start with that script and edit the code to produce the charts in this section. ::: -Beginning with the repository script, changing the value of the harmonics variable will change the complexity of the harmonic curve fit by superimposing more or fewer harmonic waves on each other. While fitting higher-order functions improves the goodness-of-fit of the model to a given set of data, many of the coefficients may be close to zero at higher numbers or harmonic terms. Fig. F4.6.10 shows the fit through the example point using one, two, and three harmonic curves. +Beginning with the repository script, changing the value of the harmonics variable will change the complexity of the harmonic curve fit by superimposing more or fewer harmonic waves on each other. While fitting higher-order functions improves the goodness-of-fit of the model to a given set of data, many of the coefficients may be close to zero at higher numbers or harmonic terms. Fig. F4.6.10 shows the fit through the example point using one, two, and three harmonic curves. ![](F4/image24.png) ![](F4/image35.png) -![](F4/image64.png) +![Fig. F4.6.10 Fit with harmonic curves of increasing complexity, fitted for data at a given point ](F4/image64.png) -Fig. F4.6.10 Fit with harmonic curves of increasing complexity, fitted for data at a given point  ## Synthesis {.unnumbered} -Assignment 1. Fit two NDVI harmonic models for a point close to Manaus, Brazil: one prior to a disturbance event and one after the disturbance event (Fig. F4.6.11). You can start with the code checkpoint below, which gives you the point coordinates and defines the initial functions needed. The disturbance event happened in mid-December 2014, so set filter dates for the first ImageCollection to '2013-01-01','2014-12-12', and set the filter dates for the second ImageCollection to '2014-12-13','2019-01-01'. Merge both fitted collections and plot both NDVI and fitted values. The result should look like Fig. F4.6.12. +Assignment 1. Fit two NDVI harmonic models for a point close to Manaus, Brazil: one prior to a disturbance event and one after the disturbance event (Fig. F4.6.11). You can start with the code checkpoint below, which gives you the point coordinates and defines the initial functions needed. The disturbance event happened in mid-December 2014, so set filter dates for the first ImageCollection to '2013-01-01','2014-12-12', and set the filter dates for the second ImageCollection to '2014-12-13','2019-01-01'. Merge both fitted collections and plot both NDVI and fitted values. The result should look like Fig. F4.6.12. -::: {.callout-note} -Code Checkpoint F46s1. The book’s repository contains a script that shows what your code should look like at this point. +:::{.callout-note} +Code Checkpoint F46s1. The book’s repository contains a script that shows what your code should look like at this point. ::: -![](F4/image26.png) +![Fig. F4.6.11 Landsat 8 images showing the land cover change at a point in Manaus, Brazil; (left) July, 6, 2014, (right) August 8, 2015](F4/image26.png) -Fig. F4.6.11 Landsat 8 images showing the land cover change at a point in Manaus, Brazil; (left) July, 6, 2014, (right) August 8, 2015 -![](F4/image36.png) +![Fig. F4.6.12 Fitted harmonic models before and after disturbance events to a given point in the Brazilian Amazon](F4/image36.png) -Fig. F4.6.12 Fitted harmonic models before and after disturbance events to a given point in the Brazilian Amazon -What do you notice? Think about how the harmonic model would look if you tried to fit the entire period. In this example, you were given the date of the breakpoint between the two conditions of the land surface within the time series. State-of-the-art land cover change algorithms work by assessing the difference between the modeled and observed pixel values. These algorithms look for breakpoints in the model, typically flagging changes after a predefined number of consecutive observations. +What do you notice? Think about how the harmonic model would look if you tried to fit the entire period. In this example, you were given the date of the breakpoint between the two conditions of the land surface within the time series. State-of-the-art land cover change algorithms work by assessing the difference between the modeled and observed pixel values. These algorithms look for breakpoints in the model, typically flagging changes after a predefined number of consecutive observations. -::: {.callout-note} -Code Checkpoint F46s2. The book’s repository contains a script that shows what your code should look like at this point. +:::{.callout-note} +Code Checkpoint F46s2. The book’s repository contains a script that shows what your code should look like at this point. ::: ## Conclusion {.unnumbered} -In this chapter, we learned how to graph and fit both linear and harmonic functions to time series of remotely sensed data. These skills underpin important tools such as Continuous Change Detection and Classification (CCDC, Chap. F4.7) and Continuous Degradation Detection (CODED, Chap. A3.4). These approaches are used by many organizations to detect forest degradation and deforestation (e.g., Tang et al. 2019, Bullock et al. 2020). These approaches can also be used to identify crops (Chap. A1.1) with high degrees of accuracy (Ghazaryan et al. 2018). +In this chapter, we learned how to graph and fit both linear and harmonic functions to time series of remotely sensed data. These skills underpin important tools such as Continuous Change Detection and Classification (CCDC, Chap. F4.7) and Continuous Degradation Detection (CODED, Chap. A3.4). These approaches are used by many organizations to detect forest degradation and deforestation (e.g., Tang et al. 2019, Bullock et al. 2020). These approaches can also be used to identify crops (Chap. A1.1) with high degrees of accuracy (Ghazaryan et al. 2018). ## References {.unnumbered} @@ -1924,12 +1950,12 @@ Tang X, Bullock EL, Olofsson P, et al (2019) Near real-time monitoring of tropic -::: {.callout-tip} +:::{.callout-tip} # Chapter Information ## Author {.unlisted .unnumbered} -  + Paulo Arévalo, Pontus Olofsson @@ -1960,7 +1986,7 @@ Continuous Change Detection and Classification (CCDC) is a land change monitorin * Work with array images (Chap. F3.1, Chap. F4.6). * Interpret fitted harmonic models (Chap. F4.6). -## Introduction to Theory  +## Introduction to Theory “A time series is a sequence of observations taken sequentially in time. … An intrinsic feature of a time series is that, typically, adjacent observations are dependent. Time-series analysis is concerned with techniques for the analysis of this dependency.” This is the formal definition of time-series analysis by Box et al. (1994). In a remote sensing context, the observations of interest are measurements of radiation reflected from the surface of the Earth from the Sun or an instrument emitting energy toward Earth. Consecutive measurements made over a given area result in a time series of surface reflectance. By analyzing such time series, we can achieve a comprehensive characterization of ecosystem and land surface processes (Kennedy et al. 2014). The result is a shift away from traditional, retrospective change-detection approaches based on data acquired over the same area at two or a few points in time to continuous monitoring of the landscape (Woodcock et al. 2020). Previous obstacles related to data storage, preprocessing, and computing power have been largely overcome with the emergence of powerful cloud-computing platforms that provide direct access to the data (Gorelick et al. 2017). In this chapter, we will illustrate how to study landscape dynamics in the Amazon river basin by analyzing dense time series of Landsat data using the CCDC algorithm. Unlike LandTrendr (Chap. F4.5), which uses anniversary images to fit straight line segments that describe the spectral trajectory over time, CCDC uses all available clear observations. This has multiple advantages, including the ability to detect changes within a year and capture seasonal patterns, although at the expense of much higher computational demands and more complexity to manipulate the outputs, compared to LandTrendr. @@ -1969,98 +1995,101 @@ Continuous Change Detection and Classification (CCDC) is a land change monitorin Spectral change is detected at the pixel level by testing for structural breaks in a time series of reflectance. In Earth Engine, this process is referred to as “temporal segmentation,” as pixel-level time series are segmented according to periods of unique reflectance. It does so by fitting harmonic regression models to all spectral bands in the time series. The model-fitting starts at the beginning of the time series and moves forward in time in an “online” approach to change detection. The coefficients are used to predict future observations, and if the residuals of future observations exceed a statistical threshold for numerous consecutive observations, then the algorithm flags that a change has occurred. After the change, a new regression model is fit and the process continues until the end of the time series. The details of the original algorithm are described in Zhu and Woodcock (2014). We have created an interface-based tool (Arévalo et al. 2020) that facilitates the exploration of time series of Landsat observations and the CCDC results. -::: {.callout-note} -Code Checkpoint F47a. The book’s repository contains information about accessing the CCDC interface. +:::{.callout-note} +Code Checkpoint F47a. The book’s repository contains information about accessing the CCDC interface. ::: -Once you have loaded the CCDC interface (Fig. F4.7.1), you will be able to navigate to any location, pick a Landsat spectral band or index to plot, and click on the map to see the fit by CCDC at the location you clicked. For this exercise, we will study landscape dynamics in the state of Rondônia, Brazil. We can use the panel on the left-bottom corner to enter the following coordinates (latitude, longitude): -9.0002, -62.7223. A point will be added in that location and the map will zoom in to it. Once there, click on the point and wait for the chart at the bottom to load. This example shows the Landsat time series for the first shortwave infrared (SWIR1) band (as blue dots) and the time segments (as colored lines) run using CCDC default parameters. The first segment represents stable forest, which was abruptly cut in mid-2006. The algorithm detects this change event and fits a new segment afterwards, representing a new temporal pattern of agriculture. Other subsequent patterns are detected as new segments are fitted that may correspond to cycles of harvest and regrowth, or a different crop. To investigate the dynamics over time, you can click on the points in the chart, and the Landsat images they correspond to will be added to the map according to the visualization parameters selected for the RGB combination in the left panel. Currently, changes made in that panel are not immediate but must be set before clicking on the map.  +Once you have loaded the CCDC interface (Fig. F4.7.1), you will be able to navigate to any location, pick a Landsat spectral band or index to plot, and click on the map to see the fit by CCDC at the location you clicked. For this exercise, we will study landscape dynamics in the state of Rondônia, Brazil. We can use the panel on the left-bottom corner to enter the following coordinates (latitude, longitude): -9.0002, -62.7223. A point will be added in that location and the map will zoom in to it. Once there, click on the point and wait for the chart at the bottom to load. This example shows the Landsat time series for the first shortwave infrared (SWIR1) band (as blue dots) and the time segments (as colored lines) run using CCDC default parameters. The first segment represents stable forest, which was abruptly cut in mid-2006. The algorithm detects this change event and fits a new segment afterwards, representing a new temporal pattern of agriculture. Other subsequent patterns are detected as new segments are fitted that may correspond to cycles of harvest and regrowth, or a different crop. To investigate the dynamics over time, you can click on the points in the chart, and the Landsat images they correspond to will be added to the map according to the visualization parameters selected for the RGB combination in the left panel. Currently, changes made in that panel are not immediate but must be set before clicking on the map. Pay special attention to the characteristics of each segment. For example, look at the average surface reflectance value for each segment. The presence of a pronounced slope may be indicative of phenomena like vegetation regrowth or degradation. The number of harmonics used in each segment may represent seasonality in vegetation (either natural or due to agricultural practices) or landscape dynamics (e.g., seasonal flooding). -![](F4/image6.png) +![Fig. 4.7.1 Landsat time series for the SWIR1 band (blue dots) and CCDC time segments (colored lines) showing a forest loss event circa 2006 for a place in Rondônia, Brazil](F4/image6.png) -Fig. 4.7.1 Landsat time series for the SWIR1 band (blue dots) and CCDC time segments (colored lines) showing a forest loss event circa 2006 for a place in Rondônia, Brazil -Question 1. While still using the SWIR1 band, click on a pixel that is forested. What do the time series and time segments look like? +Question 1. While still using the SWIR1 band, click on a pixel that is forested. What do the time series and time segments look like? ## Running CCDC -The tool shown above is useful for understanding the temporal dynamics for a specific point. However, we can do a similar analysis for larger areas by first running the CCDC algorithm over a group of pixels. The CCDC function in Earth Engine can take any ImageCollection, ideally one with little or no noise, such as a Landsat ImageCollection where clouds and cloud shadows have been masked. CCDC contains an internal cloud masking algorithm and is rather robust against missed clouds, but the cleaner the data the better. To simplify the process, we have developed a function library that contains functions for generating input data and processing CCDC results. Paste this line of code in a new script: +The tool shown above is useful for understanding the temporal dynamics for a specific point. However, we can do a similar analysis for larger areas by first running the CCDC algorithm over a group of pixels. The CCDC function in Earth Engine can take any ImageCollection, ideally one with little or no noise, such as a Landsat ImageCollection where clouds and cloud shadows have been masked. CCDC contains an internal cloud masking algorithm and is rather robust against missed clouds, but the cleaner the data the better. To simplify the process, we have developed a function library that contains functions for generating input data and processing CCDC results. Paste this line of code in a new script: -var utils = require(   'users/parevalo_bu/gee-ccdc-tools:ccdcUtilities/api'); +var utils = require( 'users/parevalo_bu/gee-ccdc-tools:ccdcUtilities/api'); -For the current exercise, we will obtain an ImageCollection of Landsat 4, 5, 7, and 8 data (Collection 2 Tier 1) that has been filtered for clouds, cloud shadows, haze, and radiometrically saturated pixels. If we were to do this manually, we would retrieve each ImageCollection for each satellite, apply the corresponding filters and then merge them all into a single ImageCollection. Instead, to simplify that process, we will use the function getLandsat, included in the “Inputs” module of our utilities, and then filter the resulting ImageCollection to a small study region for the period between 2000 and 2020. The getLandsat function will retrieve all surface reflectance bands (renamed and scaled to actual surface reflectance units) as well as other vegetation indices. To simplify the exercise, we will select only the surface reflectance bands we are going to use, adding the following code to your script: +For the current exercise, we will obtain an ImageCollection of Landsat 4, 5, 7, and 8 data (Collection 2 Tier 1) that has been filtered for clouds, cloud shadows, haze, and radiometrically saturated pixels. If we were to do this manually, we would retrieve each ImageCollection for each satellite, apply the corresponding filters and then merge them all into a single ImageCollection. Instead, to simplify that process, we will use the function getLandsat, included in the “Inputs” module of our utilities, and then filter the resulting ImageCollection to a small study region for the period between 2000 and 2020. The getLandsat function will retrieve all surface reflectance bands (renamed and scaled to actual surface reflectance units) as well as other vegetation indices. To simplify the exercise, we will select only the surface reflectance bands we are going to use, adding the following code to your script: -var studyRegion = ee.Geometry.Rectangle([ -   [-63.9533, -10.1315], -   [-64.9118, -10.6813] +var studyRegion = ee.Geometry.Rectangle([ + [-63.9533, -10.1315], + [-64.9118, -10.6813] ]); +```js // Define start, end dates and Landsat bands to use. -var startDate = '2000-01-01'; -var endDate = '2020-01-01'; -var bands = ['BLUE', 'GREEN', 'RED', 'NIR', 'SWIR1', 'SWIR2']; +var startDate = '2000-01-01'; +var endDate = '2020-01-01'; +var bands = ['BLUE', 'GREEN', 'RED', 'NIR', 'SWIR1', 'SWIR2']; // Retrieve all clear, Landsat 4, 5, 7 and 8 observations (Collection 2, Tier 1). -var filteredLandsat = utils.Inputs.getLandsat({ -       collection: 2   }) -   .filterBounds(studyRegion) -   .filterDate(startDate, endDate) -   .select(bands); +var filteredLandsat = utils.Inputs.getLandsat({ + collection: 2 }) + .filterBounds(studyRegion) + .filterDate(startDate, endDate) + .select(bands); print(filteredLandsat.first()); -With the ImageCollection ready, we can specify the CCDC parameters and run the algorithm. For this exercise we will use the default parameters, which tend to work reasonably well in most circumstances. The only parameters we will modify are the breakpoint bands, date format, and lambda. We will set all the parameter values in a dictionary that we will pass to the CCDC function. For the break detection process we use all bands except for the blue and surface temperature bands ('BLUE' and 'TEMP', respectively). The minObservations default value of 6 represents the number of consecutive observations required to flag a change. The chiSquareProbability and minNumOfYearsScaler default parameters of 0.99 and 1.33, respectively, control the sensitivity of the algorithm to detect change and the iterative curve fitting process required to detect change. We set the date format to 1, which corresponds to fractional years and tends to be easier to interpret. For instance, a change detected in the middle day of the year 2010 would be stored in a pixel as 2010.5. Finally, we use the default value of lambda of 20, but we scale it to match the scale of the inputs (surface reflectance units), and we specify a maxIterations value of 10000, instead of the default of 25000, which might take longer to complete. Those two parameters control the curve fitting process. +``` +With the ImageCollection ready, we can specify the CCDC parameters and run the algorithm. For this exercise we will use the default parameters, which tend to work reasonably well in most circumstances. The only parameters we will modify are the breakpoint bands, date format, and lambda. We will set all the parameter values in a dictionary that we will pass to the CCDC function. For the break detection process we use all bands except for the blue and surface temperature bands ('BLUE' and 'TEMP', respectively). The minObservations default value of 6 represents the number of consecutive observations required to flag a change. The chiSquareProbability and minNumOfYearsScaler default parameters of 0.99 and 1.33, respectively, control the sensitivity of the algorithm to detect change and the iterative curve fitting process required to detect change. We set the date format to 1, which corresponds to fractional years and tends to be easier to interpret. For instance, a change detected in the middle day of the year 2010 would be stored in a pixel as 2010.5. Finally, we use the default value of lambda of 20, but we scale it to match the scale of the inputs (surface reflectance units), and we specify a maxIterations value of 10000, instead of the default of 25000, which might take longer to complete. Those two parameters control the curve fitting process. -To complete the input parameters, we specify the ImageCollection to use, which we derived in the previous code section. Add this code below: +To complete the input parameters, we specify the ImageCollection to use, which we derived in the previous code section. Add this code below: +```js // Set CCD params to use. -var ccdParams = { -   breakpointBands: ['GREEN', 'RED', 'NIR', 'SWIR1', 'SWIR2'], -   tmaskBands: ['GREEN', 'SWIR2'], -   minObservations: 6, -   chiSquareProbability: 0.99, -   minNumOfYearsScaler: 1.33, -   dateFormat: 1, -   lambda: 0.002, -   maxIterations: 10000, -   collection: filteredLandsat +var ccdParams = { + breakpointBands: ['GREEN', 'RED', 'NIR', 'SWIR1', 'SWIR2'], + tmaskBands: ['GREEN', 'SWIR2'], + minObservations: 6, + chiSquareProbability: 0.99, + minNumOfYearsScaler: 1.33, + dateFormat: 1, + lambda: 0.002, + maxIterations: 10000, + collection: filteredLandsat }; // Run CCD. -var ccdResults = ee.Algorithms.TemporalSegmentation.Ccdc(ccdParams); +var ccdResults = ee.Algorithms.TemporalSegmentation.Ccdc(ccdParams); print(ccdResults); -Notice that the output ccdResults contains a large number of bands, with some of them corresponding to two-dimensional arrays. We will explore these bands more in the following section. The process of running the algorithm interactively for more than a handful of pixels can become very taxing to the system very quickly, resulting in memory errors. To avoid having such issues, we typically export the results to an Earth Engine asset first, and then inspect the asset. This approach ensures that CCDC completes its run successfully, and also allows us to access the results easily later. In the following sections of this chapter, we will use a precomputed asset, instead of asking you to export the asset yourself. For your reference, the code required to export CCDC results is shown below, with the flag set to false to help you remember to not export the results now, but instead to use the precomputed asset in the following sections. +``` +Notice that the output ccdResults contains a large number of bands, with some of them corresponding to two-dimensional arrays. We will explore these bands more in the following section. The process of running the algorithm interactively for more than a handful of pixels can become very taxing to the system very quickly, resulting in memory errors. To avoid having such issues, we typically export the results to an Earth Engine asset first, and then inspect the asset. This approach ensures that CCDC completes its run successfully, and also allows us to access the results easily later. In the following sections of this chapter, we will use a precomputed asset, instead of asking you to export the asset yourself. For your reference, the code required to export CCDC results is shown below, with the flag set to false to help you remember to not export the results now, but instead to use the precomputed asset in the following sections. -var exportResults = false -if (exportResults) {   // Create a metadata dictionary with the parameters and arguments used.   var metadata = ccdParams; -   metadata['breakpointBands'] = -       metadata['breakpointBands'].toString(); -   metadata['tmaskBands'] = metadata['tmaskBands'].toString(); -   metadata['startDate'] = startDate; -   metadata['endDate'] = endDate; -   metadata['bands'] = bands.toString();   // Export results, assigning the metadata as image properties.   //   Export.image.toAsset({ -       image: ccdResults.set(metadata), -       region: studyRegion, -       pyramidingPolicy: {           ".default": 'sample'       }, -       scale: 30   }); +var exportResults = false +if (exportResults) { // Create a metadata dictionary with the parameters and arguments used. var metadata = ccdParams; + metadata['breakpointBands'] = + metadata['breakpointBands'].toString(); + metadata['tmaskBands'] = metadata['tmaskBands'].toString(); + metadata['startDate'] = startDate; + metadata['endDate'] = endDate; + metadata['bands'] = bands.toString(); // Export results, assigning the metadata as image properties. // Export.image.toAsset({ + image: ccdResults.set(metadata), + region: studyRegion, + pyramidingPolicy: { ".default": 'sample' }, + scale: 30 }); } -Note the metadata variable above. This is not strictly required for exporting the per-pixel CCDC results, but it allows us to keep a record of important properties of the run by attaching this information as metadata to the image. Additionally, some of the tools we have created to interact with CCDC outputs use this user-created metadata to facilitate using the asset. Note also that setting the value of pyramidingPolicy to 'sample' ensures that all the bands in the output have the proper policy. +Note the metadata variable above. This is not strictly required for exporting the per-pixel CCDC results, but it allows us to keep a record of important properties of the run by attaching this information as metadata to the image. Additionally, some of the tools we have created to interact with CCDC outputs use this user-created metadata to facilitate using the asset. Note also that setting the value of pyramidingPolicy to 'sample' ensures that all the bands in the output have the proper policy. As a general rule, try to use pre-existing CCDC results if possible, and if you want to try running it yourself outside of this lab exercise, start with very small areas. For instance, the study area in this exercise would take approximately 30 minutes on average to export, but larger tiles may take several hours to complete, depending on the number of images in the collection and the parameters used. -::: {.callout-note} -Code Checkpoint F47b. The book’s repository contains a script that shows what your code should look like at this point. +:::{.callout-note} +Code Checkpoint F47b. The book’s repository contains a script that shows what your code should look like at this point. ::: ## Extracting Break Information We will now start exploring the pre-exported CCDC results mentioned in the previous section. We will make use of the third-party module palettes, described in detail in Chap. F6.0, that simplifies the use of palettes for visualization. Paste the following code in a new script: -var palettes = require('users/gena/packages:palettes'); +var palettes = require('users/gena/packages:palettes'); -var resultsPath =   'projects/gee-book/assets/F4-7/Rondonia_example_small'; -var ccdResults = ee.Image(resultsPath); +var resultsPath = 'projects/gee-book/assets/F4-7/Rondonia_example_small'; +var ccdResults = ee.Image(resultsPath); Map.centerObject(ccdResults, 10); print(ccdResults); @@ -2070,209 +2099,222 @@ The first line calls a library that will facilitate visualizing the images. The * tEnd: The end date of each time segment * tBreak: The time segment break date if a change is detected * numObs: The number of observations used in each time segment -* changeProb: A numeric value representing the change probability for each of the bands used for change detection +* changeProb: A numeric value representing the change probability for each of the bands used for change detection * *_coefs: The regression coefficients for each of the bands in the input image collection * *_rmse: The model root-mean-square error for each time segment and input band * *_magnitude: For time segments with detected changes, this represents the normalized residuals during the change period -Notice that next to the band name and band type, there is also the number of dimensions (i.e., 1 dimension, 2 dimensions). This is an indication that we are dealing with an array image, which typically requires a specific set of functions for proper manipulation, some of which we will use in the next steps. We will start by looking at the change bands, which are one of the key outputs of the CCDC algorithm. We will select the band containing the information on the timing of break, and find the number of breaks for a given time range. In the same script, paste the code below: +Notice that next to the band name and band type, there is also the number of dimensions (i.e., 1 dimension, 2 dimensions). This is an indication that we are dealing with an array image, which typically requires a specific set of functions for proper manipulation, some of which we will use in the next steps. We will start by looking at the change bands, which are one of the key outputs of the CCDC algorithm. We will select the band containing the information on the timing of break, and find the number of breaks for a given time range. In the same script, paste the code below: +```js // Select time of break and change probability array images. -var change = ccdResults.select('tBreak'); -var changeProb = ccdResults.select('changeProb'); +var change = ccdResults.select('tBreak'); +var changeProb = ccdResults.select('changeProb'); // Set the time range we want to use and get as mask of // places that meet the condition. -var start = 2000; -var end = 2021; -var mask = change.gt(start).and(change.lte(end)).and(changeProb.eq( +var start = 2000; +var end = 2021; +var mask = change.gt(start).and(change.lte(end)).and(changeProb.eq( 1)); Map.addLayer(changeProb, {}, 'change prob'); // Obtain the number of breaks for the time range. -var numBreaks = mask.arrayReduce(ee.Reducer.sum(), [0]); +var numBreaks = mask.arrayReduce(ee.Reducer.sum(), [0]); Map.addLayer(numBreaks, { -   min: 0, -   max: 5}, 'Number of breaks'); + min: 0, + max: 5}, 'Number of breaks'); -With this code, we define the time range that we want to use, and then we generate a mask that will indicate all the positions in the image array with breaks detected in that range that also meet the condition of having a change probability of 1, effectively removing some spurious breaks. For each pixel, we can count the number of times that the mask retrieved a valid result, indicating the number of breaks detected by CCDC. In the loaded layer, places that appear brighter will show a higher number of breaks, potentially indicating the conversion from forest to agriculture, followed by multiple agricultural cycles. Keep in mind that the detection of a break does not always imply a change of land cover. Natural events, small-scale disturbances and seasonal cycles, among others, can result in the detection of a break by CCDC. Similarly, changes in the condition of the land cover in a pixel can also be detected as breaks by CCDC, and some erroneous breaks can also happen due to noisy time series or other factors. +``` +With this code, we define the time range that we want to use, and then we generate a mask that will indicate all the positions in the image array with breaks detected in that range that also meet the condition of having a change probability of 1, effectively removing some spurious breaks. For each pixel, we can count the number of times that the mask retrieved a valid result, indicating the number of breaks detected by CCDC. In the loaded layer, places that appear brighter will show a higher number of breaks, potentially indicating the conversion from forest to agriculture, followed by multiple agricultural cycles. Keep in mind that the detection of a break does not always imply a change of land cover. Natural events, small-scale disturbances and seasonal cycles, among others, can result in the detection of a break by CCDC. Similarly, changes in the condition of the land cover in a pixel can also be detected as breaks by CCDC, and some erroneous breaks can also happen due to noisy time series or other factors. For places with many changes, visualizing the first or last time when a break was recorded can be helpful to understand the change dynamics happening in the landscape. Paste the code below in the same script: +```js // Obtain the first change in that time period. -var dates = change.arrayMask(mask).arrayPad([1]); -var firstChange = dates -   .arraySlice(0, 0, 1) -   .arrayFlatten([ -       ['firstChange'] -   ]) -   .selfMask(); +var dates = change.arrayMask(mask).arrayPad([1]); +var firstChange = dates + .arraySlice(0, 0, 1) + .arrayFlatten([ + ['firstChange'] + ]) + .selfMask(); -var timeVisParams = { -   palette: palettes.colorbrewer.YlOrRd[9], -   min: start, -   max: end +var timeVisParams = { + palette: palettes.colorbrewer.YlOrRd[9], + min: start, + max: end }; Map.addLayer(firstChange, timeVisParams, 'First change'); // Obtain the last change in that time period. -var lastChange = dates -   .arraySlice(0, -1) -   .arrayFlatten([ -       ['lastChange'] -   ]) -   .selfMask(); +var lastChange = dates + .arraySlice(0, -1) + .arrayFlatten([ + ['lastChange'] + ]) + .selfMask(); Map.addLayer(lastChange, timeVisParams, 'Last change'); -Here we use arrayMask to keep only the change dates that meet our condition, by using the mask we created previously. We use the function arrayPad to fill or “pad” those pixels that did not experience any change and therefore have no value in the tBreak band. Then we select either the first or last values in the array, and we convert the image from a one-dimensional array to a regular image, in order to apply a visualization to it, using a custom palette. The results should look like Fig. F4.7.2. +``` +Here we use arrayMask to keep only the change dates that meet our condition, by using the mask we created previously. We use the function arrayPad to fill or “pad” those pixels that did not experience any change and therefore have no value in the tBreak band. Then we select either the first or last values in the array, and we convert the image from a one-dimensional array to a regular image, in order to apply a visualization to it, using a custom palette. The results should look like Fig. F4.7.2. Finally, we can use the magnitude bands to visualize where and when the largest changes as recorded by CCDC have occurred, during our selected time period. We are going to use the magnitude of change in the SWIR1 band, masking it and padding it in the same way we did before. Paste this code in your script: +```js // Get masked magnitudes. -var magnitudes = ccdResults -   .select('SWIR1_magnitude') -   .arrayMask(mask) -   .arrayPad([1]); +var magnitudes = ccdResults + .select('SWIR1_magnitude') + .arrayMask(mask) + .arrayPad([1]); // Get index of max abs magnitude of change. -var maxIndex = magnitudes -   .abs() -   .arrayArgmax() -   .arrayFlatten([ -       ['index'] -   ]); +var maxIndex = magnitudes + .abs() + .arrayArgmax() + .arrayFlatten([ + ['index'] + ]); // Select max magnitude and its timing -var selectedMag = magnitudes.arrayGet(maxIndex); -var selectedTbreak = dates.arrayGet(maxIndex).selfMask(); +var selectedMag = magnitudes.arrayGet(maxIndex); +var selectedTbreak = dates.arrayGet(maxIndex).selfMask(); -var magVisParams = { -   palette: palettes.matplotlib.viridis[7], -   min: -0.15, -   max: 0.15 +var magVisParams = { + palette: palettes.matplotlib.viridis[7], + min: -0.15, + max: 0.15 }; Map.addLayer(selectedMag, magVisParams, 'Max mag'); Map.addLayer(selectedTbreak, timeVisParams, 'Time of max mag'); +``` ![](F4/image69.png) -![](F4/image1.png) +![Fig. F4.7.2 First (top) and last (bottom) detected breaks for the study area. Darker colors represent more recent dates, while brighter colors represent older dates. The first change layer shows the clear patterns of original agricultural expansion closer to the year 2000. The last change layer shows the more recently detected and noisy breaks in the same areas. The thin areas in the center of the image have only one time of change, corresponding to a single deforestation event. Pixels with no detected breaks are masked and therefore show the basemap underneath, set to show satellite imagery.](F4/image1.png) -Fig. F4.7.2 First (top) and last (bottom) detected breaks for the study area. Darker colors represent more recent dates, while brighter colors represent older dates. The first change layer shows the clear patterns of original agricultural expansion closer to the year 2000. The last change layer shows the more recently detected and noisy breaks in the same areas. The thin areas in the center of the image have only one time of change, corresponding to a single deforestation event. Pixels with no detected breaks are masked and therefore show the basemap underneath, set to show satellite imagery. We first take the absolute value because the magnitudes can be positive or negative, depending on the direction of the change and the band used. For example, a positive value in the SWIR1 may show a forest loss event, where surface reflectance goes from low to higher values. Brighter values in Fig. 4.7.3 represent events of that type. Conversely, a flooding event would have a negative value, due to the corresponding drop in reflectance. Once we find the maximum absolute value, we find its position on the array and then use that index to extract the original magnitude value, as well as the time when that break occurred. -![](F4/image44.png) +![Fig. F4.7.3 Maximum magnitude of change for the SWIR1 band for the selected study period](F4/image44.png) -Fig. F4.7.3 Maximum magnitude of change for the SWIR1 band for the selected study period -::: {.callout-note} -Code Checkpoint F47c. The book’s repository contains a script that shows what your code should look like at this point. +:::{.callout-note} +Code Checkpoint F47c. The book’s repository contains a script that shows what your code should look like at this point. ::: -Question 2. Compare the “first change” and “last change” layers with the layer showing the timing of the maximum magnitude of change. Use the Inspector to check the values for specific pixels if necessary. What does the timing of the layers tell you about the change processes happening in the area? +Question 2. Compare the “first change” and “last change” layers with the layer showing the timing of the maximum magnitude of change. Use the Inspector to check the values for specific pixels if necessary. What does the timing of the layers tell you about the change processes happening in the area? Question 3. Looking at the “max magnitude of change” layer, find places showing the largest and the smallest values. What type of changes do you think are happening in each of those places? ## Extracting Coefficients Manually -In addition to the change information generated by the CCDC algorithm, we can use the coefficients of the time segments for multiple purposes, like land cover classification. Each time segment can be described as a harmonic function with an intercept, slope, and three pairs of sine and cosine terms that allow the time segments to represent seasonality occurring at different temporal scales. These coefficients, as well as the root-mean-square error (RMSE) obtained by comparing each predicted and actual Landsat value, are produced when the CCDC algorithm is run. The following example will show you how to retrieve the intercept coefficient for a segment intersecting a specific date. In a new script, paste the code below: +In addition to the change information generated by the CCDC algorithm, we can use the coefficients of the time segments for multiple purposes, like land cover classification. Each time segment can be described as a harmonic function with an intercept, slope, and three pairs of sine and cosine terms that allow the time segments to represent seasonality occurring at different temporal scales. These coefficients, as well as the root-mean-square error (RMSE) obtained by comparing each predicted and actual Landsat value, are produced when the CCDC algorithm is run. The following example will show you how to retrieve the intercept coefficient for a segment intersecting a specific date. In a new script, paste the code below: -var palettes = require('users/gena/packages:palettes'); +var palettes = require('users/gena/packages:palettes'); -var resultsPath =   'projects/gee-book/assets/F4-7/Rondonia_example_small'; -var ccdResults = ee.Image(resultsPath); +var resultsPath = 'projects/gee-book/assets/F4-7/Rondonia_example_small'; +var ccdResults = ee.Image(resultsPath); Map.centerObject(ccdResults, 10); print(ccdResults); +```js // Display segment start and end times. -var start = ccdResults.select('tStart'); -var end = ccdResults.select('tEnd'); +var start = ccdResults.select('tStart'); +var end = ccdResults.select('tEnd'); Map.addLayer(start, { -   min: 1999, -   max: 2001}, 'Segment start'); + min: 1999, + max: 2001}, 'Segment start'); Map.addLayer(end, { -   min: 2010, -   max: 2020}, 'Segment end'); + min: 2010, + max: 2020}, 'Segment end'); -Check the Console and expand the bands section in the printed image information. We will be using the tStart, tEnd, and SWIR1_coefs bands, which are array images containing the date when the time segments start, date time segments end, and the coefficients for each of those segments for the SWIR1 band. Run the code above and switch the map to Satellite mode. Using the Inspector, click anywhere on the images, noticing the number of dates printed and their values for multiple clicked pixels. You will notice that for places with stable forest cover, there is usually one value for tStart and one for tEnd. This means that for those more stable places, only one time segment was fit by CCDC. On the other hand, for places with visible transformation in the basemap, the number of dates is usually two or three, meaning that the algorithm fitted two or three time segments, respectively. To simplify the processing of the data, we can select a single segment to extract its coefficients. Paste the code below and re-run the script: +``` +Check the Console and expand the bands section in the printed image information. We will be using the tStart, tEnd, and SWIR1_coefs bands, which are array images containing the date when the time segments start, date time segments end, and the coefficients for each of those segments for the SWIR1 band. Run the code above and switch the map to Satellite mode. Using the Inspector, click anywhere on the images, noticing the number of dates printed and their values for multiple clicked pixels. You will notice that for places with stable forest cover, there is usually one value for tStart and one for tEnd. This means that for those more stable places, only one time segment was fit by CCDC. On the other hand, for places with visible transformation in the basemap, the number of dates is usually two or three, meaning that the algorithm fitted two or three time segments, respectively. To simplify the processing of the data, we can select a single segment to extract its coefficients. Paste the code below and re-run the script: +```js // Find the segment that intersects a given date. -var targetDate = 2005.5; -var selectSegment = start.lte(targetDate).and(end.gt(targetDate)); +var targetDate = 2005.5; +var selectSegment = start.lte(targetDate).and(end.gt(targetDate)); Map.addLayer(selectSegment, {}, 'Identified segment'); -In the code above, we set a time of interest, in this case the middle of 2005, and then we find the segments that meet the condition of starting before and ending after that date. Using the Inspector again, click on different locations and verify the outputs. The segment that meets the condition will have a value of 1, and the other segments will have a value of 0. We can use this information to select the coefficients for that segment, using the code below: +``` +In the code above, we set a time of interest, in this case the middle of 2005, and then we find the segments that meet the condition of starting before and ending after that date. Using the Inspector again, click on different locations and verify the outputs. The segment that meets the condition will have a value of 1, and the other segments will have a value of 0. We can use this information to select the coefficients for that segment, using the code below: +```js // Get all coefs in the SWIR1 band. -var SWIR1Coefs = ccdResults.select('SWIR1_coefs'); +var SWIR1Coefs = ccdResults.select('SWIR1_coefs'); Map.addLayer(SWIR1Coefs, {}, 'SWIR1 coefs'); // Select only those for the segment that we identified previously. -var sliceStart = selectSegment.arrayArgmax().arrayFlatten([ -   ['index'] +var sliceStart = selectSegment.arrayArgmax().arrayFlatten([ + ['index'] ]); -var sliceEnd = sliceStart.add(1); -var selectedCoefs = SWIR1Coefs.arraySlice(0, sliceStart, sliceEnd); +var sliceEnd = sliceStart.add(1); +var selectedCoefs = SWIR1Coefs.arraySlice(0, sliceStart, sliceEnd); Map.addLayer(selectedCoefs, {}, 'Selected SWIR1 coefs'); -In the piece of code above, we first select the array image with the coefficients for the SWIR1 band. Then, using the layer that we created before, we find the position where the condition is true, and use that to extract the coefficients only for that segment. Once again, you can verify that using the Inspector tab. +``` +In the piece of code above, we first select the array image with the coefficients for the SWIR1 band. Then, using the layer that we created before, we find the position where the condition is true, and use that to extract the coefficients only for that segment. Once again, you can verify that using the Inspector tab. -Finally, what we have now is the full set of coefficients for the segment that intersects the midpoint of 2005. The coefficients are in the following order: intercept, slope, cosine 1, sine 1, cosine 2, sine 2, cosine 3, and sine 3. For this exercise we will extract the intercept coefficient (Fig. 4.7.4), which is the first element in the array, using the code below: +Finally, what we have now is the full set of coefficients for the segment that intersects the midpoint of 2005. The coefficients are in the following order: intercept, slope, cosine 1, sine 1, cosine 2, sine 2, cosine 3, and sine 3. For this exercise we will extract the intercept coefficient (Fig. 4.7.4), which is the first element in the array, using the code below: +```js // Retrieve only the intercept coefficient. -var intercept = selectedCoefs.arraySlice(1, 0, 1).arrayProject([1]); -var intVisParams = { -   palette: palettes.matplotlib.viridis[7], -   min: -6, -   max: 6 +var intercept = selectedCoefs.arraySlice(1, 0, 1).arrayProject([1]); +var intVisParams = { + palette: palettes.matplotlib.viridis[7], + min: -6, + max: 6 }; Map.addLayer(intercept.arrayFlatten([ -   ['INTP'] + ['INTP'] ]), intVisParams, 'INTP_SWIR1'); -![](F4/image78.png) +``` +![Fig. F4.7.4 Values for the intercept coefficient of the segments that start before and end after the midpoint of 2005](F4/image78.png) -Fig. F4.7.4 Values for the intercept coefficient of the segments that start before and end after the midpoint of 2005 Since we run the CCDC algorithm on Landsat surface reflectance images, intercept values should represent the average reflectance of a segment. However, if you click on the image, you will see that the values are outside of the 0–1 range. This is because the intercept is calculated by the CCDC algorithm for the origin (e.g., time 0), and not for the year we requested. In order to retrieve the adjusted intercept, as well as other coefficients, we will use a different approach. -::: {.callout-note} -Code Checkpoint F47d. The book’s repository contains a script that shows what your code should look like at this point. +:::{.callout-note} +Code Checkpoint F47d. The book’s repository contains a script that shows what your code should look like at this point. ::: Section 5. Extracting Coefficients Using External Functions The code we generated in the previous section allowed us to extract a single coefficient for a single date. However, we typically want to extract a set of multiple coefficients and bands that we can use as inputs to other workflows, such as classification. To simplify that process, we will use the same function library that we saw in Sect. 2. In this section we will extract and visualize different coefficients for a single date and produce an RGB image using the intercept coefficients for multiple spectral bands for the same date. The first step involves determining the date of interest and converting the CCDC results from array images to regular multiband images for easier manipulation and faster display. In a new script, copy the code below: +```js // Load the required libraries. -var palettes = require('users/gena/packages:palettes'); -var utils = require(   'users/parevalo_bu/gee-ccdc-tools:ccdcUtilities/api'); +var palettes = require('users/gena/packages:palettes'); +var utils = require( 'users/parevalo_bu/gee-ccdc-tools:ccdcUtilities/api'); // Load the results. -var resultsPath =   'projects/gee-book/assets/F4-7/Rondonia_example_small'; -var ccdResults = ee.Image(resultsPath); +var resultsPath = 'projects/gee-book/assets/F4-7/Rondonia_example_small'; +var ccdResults = ee.Image(resultsPath); Map.centerObject(ccdResults, 10); // Convert a date into fractional years. -var inputDate = '2005-09-25'; -var dateParams = { -   inputFormat: 3, -   inputDate: inputDate, -   outputFormat: 1 +var inputDate = '2005-09-25'; +var dateParams = { + inputFormat: 3, + inputDate: inputDate, + outputFormat: 1 }; -var formattedDate = utils.Dates.convertDate(dateParams); +var formattedDate = utils.Dates.convertDate(dateParams); // Band names originally used as inputs to the CCD algorithm. -var BANDS = ['BLUE', 'GREEN', 'RED', 'NIR', 'SWIR1', 'SWIR2']; +var BANDS = ['BLUE', 'GREEN', 'RED', 'NIR', 'SWIR1', 'SWIR2']; // Names for the time segments to retrieve. -var SEGS = ['S1', 'S2', 'S3', 'S4', 'S5', 'S6', 'S7', 'S8', 'S9',   'S10' +var SEGS = ['S1', 'S2', 'S3', 'S4', 'S5', 'S6', 'S7', 'S8', 'S9', 'S10' ]; // Transform CCD results into a multiband image. -var ccdImage = utils.CCDC.buildCcdImage(ccdResults, SEGS.length, -   BANDS); +var ccdImage = utils.CCDC.buildCcdImage(ccdResults, SEGS.length, + BANDS); print(ccdImage); +``` In the code above we define the date of interest (2005-09-25) and convert it to the date format in which we ran CCDC, which corresponds to fractional years. After that, we specify the band that we used as inputs for the CCDC algorithm. Finally, we specify the names we will assign to the time segments, with the list length indicating the maximum number of time segments to retrieve per pixel. This step is done because the results generated by CCDC are stored as variable-length arrays. For example, a pixel where there are no breaks detected will have one time segment, but another pixel where a single break was detected may have one or two segments, depending on when the break occurred. Requesting a pre-defined maximum number of segments ensures that the structure of the multi-band image is known, and greatly facilitates its manipulation and display. Once we have set these variables, we call a function that converts the result into an image with several bands representing the combination of segments requested, input bands, and coefficients. You can see the image structure in the Console. Finally, to extract a subset of coefficients for the desired bands, we can use a function in the imported library, called getMultiCoefs. This function expects the following ordered parameters: @@ -2280,63 +2322,63 @@ Finally, to extract a subset of coefficients for the desired bands, we can use a * The CCDC results in the multiband format we just generated in the step above. * The date for which we want to extract the coefficients, in the format in which the CCDC results were run (fractional years in our case). * List of the bands to retrieve (i.e., spectral bands). -* List of coefficients to retrieve, defined as follows: INTP (intercept), SLP (slope), COS, SIN,COS32, SIN2, COS3, SIN3, and RMSE. -* A Boolean flag of true or false, indicating whether we want the intercepts to be calculated for the input date, instead of being calculated at the origin. If true, SLP must be included in the list of coefficients to retrieve. +* List of coefficients to retrieve, defined as follows: INTP (intercept), SLP (slope), COS, SIN,COS32, SIN2, COS3, SIN3, and RMSE. +* A Boolean flag of true or false, indicating whether we want the intercepts to be calculated for the input date, instead of being calculated at the origin. If true, SLP must be included in the list of coefficients to retrieve. * List of segment names, as used to create the multiband image in the prior step. -* Behavior to apply if there is no time segment for the requested date: normal will retrieve a value only if the date intersects a segment;        before or after will use the value of the segment immediately before or after the requested date, if no segment intersects the date directly. +* Behavior to apply if there is no time segment for the requested date: normal will retrieve a value only if the date intersects a segment; before or after will use the value of the segment immediately before or after the requested date, if no segment intersects the date directly. +```js // Define bands to select. -var SELECT_BANDS = ['RED', 'GREEN', 'BLUE', 'NIR']; +var SELECT_BANDS = ['RED', 'GREEN', 'BLUE', 'NIR']; // Define coefficients to select. // This list contains all possible coefficients, and the RMSE -var SELECT_COEFS = ['INTP', 'SLP', 'RMSE']; +var SELECT_COEFS = ['INTP', 'SLP', 'RMSE']; // Obtain coefficients. -var coefs = utils.CCDC.getMultiCoefs( -   ccdImage, formattedDate, SELECT_BANDS, SELECT_COEFS, true, -   SEGS, 'after'); +var coefs = utils.CCDC.getMultiCoefs( + ccdImage, formattedDate, SELECT_BANDS, SELECT_COEFS, true, + SEGS, 'after'); print(coefs); // Show a single coefficient. -var slpVisParams = { -   palette: palettes.matplotlib.viridis[7], -   min: -0.0005, -   max: 0.005 +var slpVisParams = { + palette: palettes.matplotlib.viridis[7], + min: -0.0005, + max: 0.005 }; -Map.addLayer(coefs.select('RED_SLP'), slpVisParams,   'RED SLOPE 2005-09-25'); +Map.addLayer(coefs.select('RED_SLP'), slpVisParams, 'RED SLOPE 2005-09-25'); -var rmseVisParams = { -   palette: palettes.matplotlib.viridis[7], -   min: 0, -   max: 0.1 +var rmseVisParams = { + palette: palettes.matplotlib.viridis[7], + min: 0, + max: 0.1 }; -Map.addLayer(coefs.select('NIR_RMSE'), rmseVisParams,   'NIR RMSE 2005-09-25'); +Map.addLayer(coefs.select('NIR_RMSE'), rmseVisParams, 'NIR RMSE 2005-09-25'); // Show an RGB with three coefficients. -var rgbVisParams = { -   bands: ['RED_INTP', 'GREEN_INTP', 'BLUE_INTP'], -   min: 0, -   max: 0.1 +var rgbVisParams = { + bands: ['RED_INTP', 'GREEN_INTP', 'BLUE_INTP'], + min: 0, + max: 0.1 }; Map.addLayer(coefs, rgbVisParams, 'RGB 2005-09-25'); -The slope and RMSE images are shown in Fig. 4.7.5. For the slopes, high positive values are bright, while large negative values are very dark. Most of the remaining forest is stable and has a slope close to zero, while areas that have experienced transformation and show agricultural activity tend to have positive slopes in the RED band, appearing bright in the image. Similarly, for the RMSE image, stable forests present more predictable time series of surface reflectance that are captured more faithfully by the time segments, and therefore present lower RMSE values, appearing darker in the image. Agricultural areas present noisier time series that are more challenging to model, and result in higher RMSE values, appearing brighter. +``` +The slope and RMSE images are shown in Fig. 4.7.5. For the slopes, high positive values are bright, while large negative values are very dark. Most of the remaining forest is stable and has a slope close to zero, while areas that have experienced transformation and show agricultural activity tend to have positive slopes in the RED band, appearing bright in the image. Similarly, for the RMSE image, stable forests present more predictable time series of surface reflectance that are captured more faithfully by the time segments, and therefore present lower RMSE values, appearing darker in the image. Agricultural areas present noisier time series that are more challenging to model, and result in higher RMSE values, appearing brighter. ![](F4/image18.png) -![](F4/image28.png) +![Fig. F4.7.5 Image showing the slopes (top) and RMSE (bottom) of the segments that intersect the requested date](F4/image28.png) -Fig. F4.7.5 Image showing the slopes (top) and RMSE (bottom) of the segments that intersect the requested date -Finally, the RGB image we created is shown in Fig. 4.7.6. The intercepts are calculated for the middle point of the time segment intercepting the date we requested, representing the average reflectance for the span of the selected segment. In that sense, when shown together as an RGB image, they are similar to a composite image for the selected date, with the advantage of always being cloud-free. +Finally, the RGB image we created is shown in Fig. 4.7.6. The intercepts are calculated for the middle point of the time segment intercepting the date we requested, representing the average reflectance for the span of the selected segment. In that sense, when shown together as an RGB image, they are similar to a composite image for the selected date, with the advantage of always being cloud-free. -![](F4/image76.png) +![Fig. F4.7.6 RGB image created using the time segment intercepts for the requested date](F4/image76.png) -Fig. F4.7.6 RGB image created using the time segment intercepts for the requested date -::: {.callout-note} -Code Checkpoint F47e. The book’s repository contains a script that shows what your code should look like at this point. +:::{.callout-note} +Code Checkpoint F47e. The book’s repository contains a script that shows what your code should look like at this point. ::: ## Synthesis {.unnumbered} @@ -2344,11 +2386,11 @@ Assignment 1. Use the time series from the first section of this chapter to expl Assignment 2. Pick three periods within the temporal study period of the CCDC results we used earlier: one near to the start, another in the middle, and the third close to the end. For each period, visualize the maximum change magnitude. Compare the spatial patterns between periods, and reflect on the types of disturbances that might be happening at each stage. -Assignment 3. Select the intercept coefficients of the middle date of each of the periods you chose in the previous assignment. For each of those dates, load an RGB image with the band combination of your choosing (or simply use the Red, Green and Blue intercepts to obtain true-color images). Using the Inspector tab, compare the values across images in places with subtle and large differences between them, as well as in areas that do not change. What do the values tell you in terms of the benefits of using CCDC to study changes in a landscape? +Assignment 3. Select the intercept coefficients of the middle date of each of the periods you chose in the previous assignment. For each of those dates, load an RGB image with the band combination of your choosing (or simply use the Red, Green and Blue intercepts to obtain true-color images). Using the Inspector tab, compare the values across images in places with subtle and large differences between them, as well as in areas that do not change. What do the values tell you in terms of the benefits of using CCDC to study changes in a landscape? ## Conclusion {.unnumbered} -This chapter provided a guide for the interpretation of the results from the CCDC algorithm for studying deforestation in the Amazon. Consider the advantages of such an analysis compared to traditional approaches to change detection, which are typically based on the comparison of two or a few images collected over the same area. For example, with time-series analysis, we can study trends and subtle processes such as vegetation recovery or degradation, determine the timing of land-surface events, and move away from retrospective analyses to monitoring in near-real time. Through the use of all available clear observations, CCDC can detect intra-annual breaks and capture seasonal patterns, although at the expense of increased computational requirements and complexity, unlike faster and easier to interpret methods based on annual composites, such as LandTrendr (Chap. F4.5). We expect to see more applications that make use of multiple change detection approaches (also known as “Ensemble” approaches), and multisensor analyses in which data from different satellites are fused (radar and optical, for example) for higher data density. +This chapter provided a guide for the interpretation of the results from the CCDC algorithm for studying deforestation in the Amazon. Consider the advantages of such an analysis compared to traditional approaches to change detection, which are typically based on the comparison of two or a few images collected over the same area. For example, with time-series analysis, we can study trends and subtle processes such as vegetation recovery or degradation, determine the timing of land-surface events, and move away from retrospective analyses to monitoring in near-real time. Through the use of all available clear observations, CCDC can detect intra-annual breaks and capture seasonal patterns, although at the expense of increased computational requirements and complexity, unlike faster and easier to interpret methods based on annual composites, such as LandTrendr (Chap. F4.5). We expect to see more applications that make use of multiple change detection approaches (also known as “Ensemble” approaches), and multisensor analyses in which data from different satellites are fused (radar and optical, for example) for higher data density. ## References {.unnumbered} @@ -2372,7 +2414,7 @@ Zhu Z, Woodcock CE (2014) Continuous change detection and classification of land -::: {.callout-tip} +:::{.callout-tip} # Chapter Information ## Author {.unlisted .unnumbered} @@ -2384,9 +2426,9 @@ Jeffrey A. Cardille, Rylan Boothman, Mary Villamor, Elijah Perez, Eidan Willis, ## Overview {.unlisted .unnumbered} -  + -As the ability to rapidly produce classifications of satellite images grows, it will be increasingly important to have algorithms that can sift through them to separate the signal from inevitable classification noise. The purpose of this chapter is to explore how to update classification time series by blending information from multiple classifications made from a wide variety of data sources. In this lab, we will explore how to update the classification time series of the Roosevelt River found in Fortin et al. (2020). That time series began with the 1972 launch of Landsat 1, blending evidence from 10 sensors and more than 140 images to show the evolution of the area until 2016. How has it changed since 2016? What new tools and data streams might we tap to understand the land surface through time? +As the ability to rapidly produce classifications of satellite images grows, it will be increasingly important to have algorithms that can sift through them to separate the signal from inevitable classification noise. The purpose of this chapter is to explore how to update classification time series by blending information from multiple classifications made from a wide variety of data sources. In this lab, we will explore how to update the classification time series of the Roosevelt River found in Fortin et al. (2020). That time series began with the 1972 launch of Landsat 1, blending evidence from 10 sensors and more than 140 images to show the evolution of the area until 2016. How has it changed since 2016? What new tools and data streams might we tap to understand the land surface through time? ## Learning Outcomes {.unlisted .unnumbered} @@ -2400,31 +2442,31 @@ As the ability to rapidly produce classifications of satellite images grows, it * Import images and image collections, filter, and visualize (Part F1). * Perform basic image analysis: select bands, compute indices, create masks, classify images (Part F2). -* ​​Create a graph using ui.Chart (Chap. F1.3). +* ​​Create a graph using ui.Chart (Chap. F1.3). * Obtain accuracy metrics from classifications (Chap. F2.2). -::: +::: ## Introduction {.unlisted .unnumbered} When working with multiple sensors, we are often presented with a challenge: What to do with classification noise? It’s almost impossible to remove all noise from a classification. Given the information contained in a stream of classifications, however, you should be able to use the temporal context to distinguish noise from true changes in the landscape. -The Bayesian Updating of Land Cover (BULC) algorithm (Cardille and Fortin 2016) is designed to extract the signal from the noise in a stream of classifications made from any number of data sources. BULC’s principal job is to estimate, at each time step, the likeliest state of land use and land cover (LULC) in a study area given the accumulated evidence to that point. It takes a stack of provisional classifications as input; in keeping with the terminology of Bayesian statistics, these are referred to as “Events,” because they provide new evidence to the system. BULC then returns a stack of classifications as output that represents the estimated LULC time series implied by the Events.  +The Bayesian Updating of Land Cover (BULC) algorithm (Cardille and Fortin 2016) is designed to extract the signal from the noise in a stream of classifications made from any number of data sources. BULC’s principal job is to estimate, at each time step, the likeliest state of land use and land cover (LULC) in a study area given the accumulated evidence to that point. It takes a stack of provisional classifications as input; in keeping with the terminology of Bayesian statistics, these are referred to as “Events,” because they provide new evidence to the system. BULC then returns a stack of classifications as output that represents the estimated LULC time series implied by the Events. BULC estimates, at each time step, the most likely class from a set given the evidence up to that point in time. This is done by employing an accuracy assessment matrix like that seen in Chap. F2.2. At each time step, the algorithm quantifies the agreement between two classifications adjacent in time within a time series. -If the Events agree strongly, they are evidence of the true condition of the landscape at that point in time. If two adjacent Events disagree, the accuracy assessment matrix limits their power to change the class of a pixel in the interpreted time series. As each new classification is processed, BULC judges the credibility of a pixel’s stated class and keeps track of a set of estimates of the probability of each class for each pixel. In this way, each pixel traces its own LULC history, reflected through BULC’s judgment of the confidence in each of the classifications. The specific mechanics and formulas of BULC are detailed in Cardille and Fortin (2016). +If the Events agree strongly, they are evidence of the true condition of the landscape at that point in time. If two adjacent Events disagree, the accuracy assessment matrix limits their power to change the class of a pixel in the interpreted time series. As each new classification is processed, BULC judges the credibility of a pixel’s stated class and keeps track of a set of estimates of the probability of each class for each pixel. In this way, each pixel traces its own LULC history, reflected through BULC’s judgment of the confidence in each of the classifications. The specific mechanics and formulas of BULC are detailed in Cardille and Fortin (2016). -BULC’s code is written in JavaScript, with modules that weigh evidence for and against change in several ways, while recording parts of the data-weighing process for you to inspect. In this lab, we will explore BULC through its graphical user interface (GUI), which allows rapid interaction with the algorithm’s main functionality. +BULC’s code is written in JavaScript, with modules that weigh evidence for and against change in several ways, while recording parts of the data-weighing process for you to inspect. In this lab, we will explore BULC through its graphical user interface (GUI), which allows rapid interaction with the algorithm’s main functionality. ## Imagery and Classifications of the Roosevelt River -How has the Roosevelt River area changed in recent decades? One way to view the area’s recent history is to use Google Earth Timelapse, which shows selected annual clear images of every part of Earth’s terrestrial surface since the 1980s. (You can find the site quickly with a web search.) Enter “Roosevelt River, Brazil” in the search field. For centuries, this area was very remote from agricultural development. It was so little known to Westerners that when former US President Theodore Roosevelt traversed it in the early 1900s there was widespread doubt about whether his near-death experience there was exaggerated or even entirely fictional (Millard 2006). After World War II, the region saw increased agricultural development. Fortin et al. (2020) traced four decades of the history of this region with satellite imagery. Timelapse, meanwhile, indicates that land cover conversion continued after 2016. Can we track it using Earth Engine? +How has the Roosevelt River area changed in recent decades? One way to view the area’s recent history is to use Google Earth Timelapse, which shows selected annual clear images of every part of Earth’s terrestrial surface since the 1980s. (You can find the site quickly with a web search.) Enter “Roosevelt River, Brazil” in the search field. For centuries, this area was very remote from agricultural development. It was so little known to Westerners that when former US President Theodore Roosevelt traversed it in the early 1900s there was widespread doubt about whether his near-death experience there was exaggerated or even entirely fictional (Millard 2006). After World War II, the region saw increased agricultural development. Fortin et al. (2020) traced four decades of the history of this region with satellite imagery. Timelapse, meanwhile, indicates that land cover conversion continued after 2016. Can we track it using Earth Engine? -In this section, we will view the classification inputs to BULC, which were made separately from this lab exercise by identifying training points and classifying them using Earth Engine’s regression tree capability. As seen in Table 4.8.1, the classification inputs included Sentinel-2 optical data, Landsat 7, Landsat 8, and the Advanced Spaceborne Thermal Emission and Reflection Radiometer (ASTER) aboard Terra. Though each classification was made with care, they each contain noise, with each pixel likely to have been misclassified one or more times. This could lead us to draw unrealistic conclusions if the classifications themselves were considered as a time series. For example, we would judge it highly unlikely that an area represented by a pixel would really be agriculture one day and revert to intact forest later in the month, only to be converted to agriculture again soon after, and so on. With careful (though unavoidably imperfect) classifications, we would expect that an area that had truly been converted to agriculture would consistently be classified as agriculture, while an area that remained as forest would be classified as that class most of the time. BULC’s logic is to detect that persistence, extracting the true LULC change and stability from the noisy signal of the time series of classifications. +In this section, we will view the classification inputs to BULC, which were made separately from this lab exercise by identifying training points and classifying them using Earth Engine’s regression tree capability. As seen in Table 4.8.1, the classification inputs included Sentinel-2 optical data, Landsat 7, Landsat 8, and the Advanced Spaceborne Thermal Emission and Reflection Radiometer (ASTER) aboard Terra. Though each classification was made with care, they each contain noise, with each pixel likely to have been misclassified one or more times. This could lead us to draw unrealistic conclusions if the classifications themselves were considered as a time series. For example, we would judge it highly unlikely that an area represented by a pixel would really be agriculture one day and revert to intact forest later in the month, only to be converted to agriculture again soon after, and so on. With careful (though unavoidably imperfect) classifications, we would expect that an area that had truly been converted to agriculture would consistently be classified as agriculture, while an area that remained as forest would be classified as that class most of the time. BULC’s logic is to detect that persistence, extracting the true LULC change and stability from the noisy signal of the time series of classifications. -Table F4.8.1 Images classified for updating Roosevelt River LULC with BULC +Table F4.8.1 Images classified for updating Roosevelt River LULC with BULC Sensor @@ -2472,106 +2514,101 @@ ASTER 15m–30m -As you have seen in earlier chapters, creating classifications can be very involved and time consuming. To allow you to concentrate on BULC’s efforts to clean noise from an existing ImageCollection, we have created the classifications already and stored them as an ImageCollection asset. You can view the Event time series using the ui.Thumbnail function, which creates an animation of the elements of the collection​​​​. Paste the code below into a new script to see those classifications drawn in sequence in the Console. +As you have seen in earlier chapters, creating classifications can be very involved and time consuming. To allow you to concentrate on BULC’s efforts to clean noise from an existing ImageCollection, we have created the classifications already and stored them as an ImageCollection asset. You can view the Event time series using the ui.Thumbnail function, which creates an animation of the elements of the collection​​​​. Paste the code below into a new script to see those classifications drawn in sequence in the Console. -var events = ee.ImageCollection(   'projects/gee-book/assets/F4-8/cleanEvents'); +var events = ee.ImageCollection( 'projects/gee-book/assets/F4-8/cleanEvents'); print(events, 'List of Events'); print('Number of events:', events.size()); print(ui.Thumbnail(events, { -   min: 0, -   max: 3, -   palette: ['black', 'green', 'blue', 'yellow'], -   framesPerSecond: 1, -   dimensions: 1000 + min: 0, + max: 3, + palette: ['black', 'green', 'blue', 'yellow'], + framesPerSecond: 1, + dimensions: 1000 })); In the thumbnail sequence, the color palette shows Forest (class 1) as green, Water (class 2) as blue, and Active Agriculture (class 3) as yellow. Areas with no data in a particular Event are shown in black. -::: {.callout-note} -Code Checkpoint F48a. The book’s repository contains a script that shows what your code should look like at this point. +:::{.callout-note} +Code Checkpoint F48a. The book’s repository contains a script that shows what your code should look like at this point. ::: ## Basics of the BULC Interface -To see if BULC can successfully sift through these Events, we will use BULC’s GUI (Fig. F4.8.1), which makes interacting with the functionality straightforward. ::: {.callout-note} -Code Checkpoint F48b in the book’s repository contains information about accessing that interface. +To see if BULC can successfully sift through these Events, we will use BULC’s GUI (Fig. F4.8.1), which makes interacting with the functionality straightforward. :::{.callout-note} +Code Checkpoint F48b in the book’s repository contains information about accessing that interface. ::: -![](F4/image5.png) +![Fig. F4.8.1 BULC interface](F4/image5.png) -Fig. F4.8.1 BULC interface -After you have run the script, BULC’s interface requires that a few parameters be set; these are specified using the left panel. Here, we describe and populate each of the required parameters, which are shown in red. As you proceed, the default red color will change to green when a parameter receives a value. +After you have run the script, BULC’s interface requires that a few parameters be set; these are specified using the left panel. Here, we describe and populate each of the required parameters, which are shown in red. As you proceed, the default red color will change to green when a parameter receives a value. -* The interface permits new runs to be created using the Manual or Automated methods. The Automated setting allows information from a previous run to be used without manual entry. In this tutorial, we will enter each parameter individually using the interface, so you should set this item to Manual by clicking once on it. -* Select type of image: The interface can accept pre-made Event inputs in one of three forms: (1) as a stored ImageCollection; (2) as a single multi-banded Image; and (3) as a stream of Dynamic World classifications. The classifications are processed in the order they are given, either within the ImageCollection or sequentially through the Image bands. For this run, select Image Collection from the dropdown menu, then enter the path to this collection, without enclosing it in quotes: projects/gee-book/assets/F4-8/cleanEvents +* The interface permits new runs to be created using the Manual or Automated methods. The Automated setting allows information from a previous run to be used without manual entry. In this tutorial, we will enter each parameter individually using the interface, so you should set this item to Manual by clicking once on it. +* Select type of image: The interface can accept pre-made Event inputs in one of three forms: (1) as a stored ImageCollection; (2) as a single multi-banded Image; and (3) as a stream of Dynamic World classifications. The classifications are processed in the order they are given, either within the ImageCollection or sequentially through the Image bands. For this run, select Image Collection from the dropdown menu, then enter the path to this collection, without enclosing it in quotes: projects/gee-book/assets/F4-8/cleanEvents * Remap: in some settings, you might want to remap the input value to combine classes. Leave this empty for now; an example of this is discussed later in the lab. -* Number of Classes in Events and Number of Classes to Track: The algorithm requires the number of classes in each Event and the number of meaningful classes to track to be entered. Here, there are 3 classes in each classification (Forest, Water, and Active Agriculture) and 3 classes being tracked. (In the BULC-U version of the algorithm (Lee et al. 2018, 2020), these numbers may be different when the Events are made using unsupervised classifications, which may contain many more classes than are being tracked in a given run.) Meaningful classes are assumed by BULC to begin with 1 rather than 0, while class values of 0 in Events are treated as no data. As seen in the thumbnail of Events, there are 3 classes; set both of these values to 3. -* The Default Study Area is used by BULC to delimit the location to analyze. This value can be pulled from a specially sized Image or set automatically, using the extent of the inputs. Set this parameter to Event Geometry, which gets the value automatically from the Event collection. -* The Base Land Cover Image defines the initial land cover condition to which BULC adds evidence from Events. Here, we are working to update the land cover map from the end of 2016, as estimated in Fortin et al. (2020). The ending estimated classification from that study has been loaded as an asset and placed as the first image in the input ImageCollection. We will direct the BULC interface to use this first image in the collection as the base land cover image by selecting Top. -* Overlay Approach: BULC can run in multiple modes, which affect the outcome of the classification updating. One option, Overlay, overlays each consecutive Event with the one prior in the sequence, following Fortin et al. (2016). Another option, Custom, allows a user-defined constant array to be used. For this tutorial, we will choose D (Identity matrix), which uses the same transition table for every Event, regardless of how it overlays with the Event prior. That table gives a large conditional likelihood to the chance that classes agree strongly across the consecutive Event classifications that are used as inputs. +* Number of Classes in Events and Number of Classes to Track: The algorithm requires the number of classes in each Event and the number of meaningful classes to track to be entered. Here, there are 3 classes in each classification (Forest, Water, and Active Agriculture) and 3 classes being tracked. (In the BULC-U version of the algorithm (Lee et al. 2018, 2020), these numbers may be different when the Events are made using unsupervised classifications, which may contain many more classes than are being tracked in a given run.) Meaningful classes are assumed by BULC to begin with 1 rather than 0, while class values of 0 in Events are treated as no data. As seen in the thumbnail of Events, there are 3 classes; set both of these values to 3. +* The Default Study Area is used by BULC to delimit the location to analyze. This value can be pulled from a specially sized Image or set automatically, using the extent of the inputs. Set this parameter to Event Geometry, which gets the value automatically from the Event collection. +* The Base Land Cover Image defines the initial land cover condition to which BULC adds evidence from Events. Here, we are working to update the land cover map from the end of 2016, as estimated in Fortin et al. (2020). The ending estimated classification from that study has been loaded as an asset and placed as the first image in the input ImageCollection. We will direct the BULC interface to use this first image in the collection as the base land cover image by selecting Top. +* Overlay Approach: BULC can run in multiple modes, which affect the outcome of the classification updating. One option, Overlay, overlays each consecutive Event with the one prior in the sequence, following Fortin et al. (2016). Another option, Custom, allows a user-defined constant array to be used. For this tutorial, we will choose D (Identity matrix), which uses the same transition table for every Event, regardless of how it overlays with the Event prior. That table gives a large conditional likelihood to the chance that classes agree strongly across the consecutive Event classifications that are used as inputs. -BULC makes relatively small demands on memory since its arithmetic uses only multiplication, addition, and division, without the need for complex function fitting. The specific memory use is tied to the overlay method used. In particular, Event-by-Event comparisons (the Overlay setting) are considerably more computationally expensive than pre-defined transition tables (the Identity and Custom settings). The maximum working Event depth is also slightly lowered when intermediate probability values are returned for inspection. Our tests indicate that with pre-defined truth tables and no intermediate probability values returned, BULC can handle updating problems hundreds of Events deep across an arbitrarily large area. +BULC makes relatively small demands on memory since its arithmetic uses only multiplication, addition, and division, without the need for complex function fitting. The specific memory use is tied to the overlay method used. In particular, Event-by-Event comparisons (the Overlay setting) are considerably more computationally expensive than pre-defined transition tables (the Identity and Custom settings). The maximum working Event depth is also slightly lowered when intermediate probability values are returned for inspection. Our tests indicate that with pre-defined truth tables and no intermediate probability values returned, BULC can handle updating problems hundreds of Events deep across an arbitrarily large area. -* Initialization Approach: If a BULC run of the full intended size ever surpassed the memory available, you would be able to break the processing into two or more parts using the following technique. First, create a slightly smaller run that can complete, and save the final probability image of that run. Because of the operation of Bayes’ theorem, the ending probability multi-band image can be used as the prior probability to continue processing Events with the same answers as if it had been run all at once. For this small run, we will select F (First Run). -* Levelers: BULC uses three levelers as part of its processing, as described in Fortin et al. (2016). The Initialization Leveler creates the initial probability vector of the initial LULC image; the Transition Leveler dampens transitions at each time step, making BULC less reactive to new evidence; and the Posterior Leveler ensures that each class retains nonzero probability so that the Bayes formula can function properly throughout the run. For this run, set the parameters to 0.65, 0.3, and 0.6, respectively. This corresponds to a typical set of values that is appropriate when moderate-quality classifications are fed to BULC. -* Colour Output Palette: We will use the same color palette as what was seen in the small script you used to draw the Events, with one exception. Because BULC will give a value for the estimated class for every pixel, there are no pixels in the study area with missing or masked data. To line up the colors with the attainable numbers, we will remove the color ‘black’ from the specification. For this field, enter this list: ['green', 'blue', 'yellow']. For all of the text inputs, make sure to click outside that field after entering text so that the input information is registered; the changing of the text color to green confirms that the information was received. +* Initialization Approach: If a BULC run of the full intended size ever surpassed the memory available, you would be able to break the processing into two or more parts using the following technique. First, create a slightly smaller run that can complete, and save the final probability image of that run. Because of the operation of Bayes’ theorem, the ending probability multi-band image can be used as the prior probability to continue processing Events with the same answers as if it had been run all at once. For this small run, we will select F (First Run). +* Levelers: BULC uses three levelers as part of its processing, as described in Fortin et al. (2016). The Initialization Leveler creates the initial probability vector of the initial LULC image; the Transition Leveler dampens transitions at each time step, making BULC less reactive to new evidence; and the Posterior Leveler ensures that each class retains nonzero probability so that the Bayes formula can function properly throughout the run. For this run, set the parameters to 0.65, 0.3, and 0.6, respectively. This corresponds to a typical set of values that is appropriate when moderate-quality classifications are fed to BULC. +* Colour Output Palette: We will use the same color palette as what was seen in the small script you used to draw the Events, with one exception. Because BULC will give a value for the estimated class for every pixel, there are no pixels in the study area with missing or masked data. To line up the colors with the attainable numbers, we will remove the color ‘black’ from the specification. For this field, enter this list: ['green', 'blue', 'yellow']. For all of the text inputs, make sure to click outside that field after entering text so that the input information is registered; the changing of the text color to green confirms that the information was received. When you have finished setting the required parameters, the interface will look like Fig. 4.8.2. -![](F4/image62.png) +![Fig. 4.8.2 Initial settings for the key driving parameters of BULC ](F4/image62.png) -Fig. 4.8.2 Initial settings for the key driving parameters of BULC  Beneath the required parameters is a set of optional parameters that affect which intermediate results are stored during a run for later inspection. We are also given a choice of returning intermediate results for closer inspection. At this stage, you can leave all optional parameters out of the BULC call by leaving them blanked or unchecked. -After clicking the Apply Parameters button at the bottom of the left panel, the classifications and parameters are sent to the BULC modules. The Map will move to the study area, and after a few seconds, the Console will hold new thumbnails. The uppermost thumbnail is a rapidly changing view of the input classifications. Beneath that is a thumbnail of the same area as interpreted by BULC. Beneath those is a Confidence thumbnail, which is discussed in detail later in this lab. +After clicking the Apply Parameters button at the bottom of the left panel, the classifications and parameters are sent to the BULC modules. The Map will move to the study area, and after a few seconds, the Console will hold new thumbnails. The uppermost thumbnail is a rapidly changing view of the input classifications. Beneath that is a thumbnail of the same area as interpreted by BULC. Beneath those is a Confidence thumbnail, which is discussed in detail later in this lab. -The BULC interpretation of the landscape looks roughly like the Event inputs, but it is different in two important ways. First, depending on the leveler settings, it will usually have less noise than the Event classifications. In the settings above, we used the Transition and Posterior levelers to tell BULC to trust past accumulated evidence more than a single new image. The second key difference between the BULC result and the input classifications is that even when the inputs don’t cover the whole area at each time step, BULC provides an estimate in every pixel at each time step. To create this continuous classification, if a new classification does not have data for some part of the study area (beyond the edge of a given image, for example), the last best guess from the previous iteration is carried forward. Simply put, the estimate in a given pixel is kept the same until new data arrives. +The BULC interpretation of the landscape looks roughly like the Event inputs, but it is different in two important ways. First, depending on the leveler settings, it will usually have less noise than the Event classifications. In the settings above, we used the Transition and Posterior levelers to tell BULC to trust past accumulated evidence more than a single new image. The second key difference between the BULC result and the input classifications is that even when the inputs don’t cover the whole area at each time step, BULC provides an estimate in every pixel at each time step. To create this continuous classification, if a new classification does not have data for some part of the study area (beyond the edge of a given image, for example), the last best guess from the previous iteration is carried forward. Simply put, the estimate in a given pixel is kept the same until new data arrives. -Meanwhile, below the Console, the rest of the interface changes when BULC is run. The Map panel displays BULC’s classification for the final date: that is, after considering the evidence from each of the input classifications. We can use the Satellite background to judge whether BULC is accurately capturing the state of LULC. This can be done by unselecting the drawn layers in the map layer set and selecting Satellite from the choices in the upper-right part of the Map panel. Earth Engine’s background satellite images are often updated, so you should see something like the right side of Fig. F4.8.3, though it may differ slightly.  +Meanwhile, below the Console, the rest of the interface changes when BULC is run. The Map panel displays BULC’s classification for the final date: that is, after considering the evidence from each of the input classifications. We can use the Satellite background to judge whether BULC is accurately capturing the state of LULC. This can be done by unselecting the drawn layers in the map layer set and selecting Satellite from the choices in the upper-right part of the Map panel. Earth Engine’s background satellite images are often updated, so you should see something like the right side of Fig. F4.8.3, though it may differ slightly. ![](F4/image57.png) -![](F4/image65.png) +![Fig. 4.8.3 BULC estimation of the state of LULC at the end of 2021 (left). Satellite backdrop for Earth Engine (right), which may differ from what you see due to updates. ](F4/image65.png) -Fig. 4.8.3 BULC estimation of the state of LULC at the end of 2021 (left). Satellite backdrop for Earth Engine (right), which may differ from what you see due to updates.   -Question 1. When comparing the BULC classification for 2021 against the current Earth Engine satellite view, what are the similarities and differences? Note that in Earth Engine, the copyrighted year numbers at the bottom of the screen may not coincide with the precise date of the image shown. +Question 1. When comparing the BULC classification for 2021 against the current Earth Engine satellite view, what are the similarities and differences? Note that in Earth Engine, the copyrighted year numbers at the bottom of the screen may not coincide with the precise date of the image shown. In the rightmost panel below the Console, the interface offers you multiple options for viewing the results. These include: -1. Movie. This uses the ui.Thumbnail API function to draw the BULC results rapidly in the viewer.  This option offers you a control on the frame rate (in frames per second), and a checkbox affecting the drawing resolution. The high-resolution option uses the maximum resolution permitted given the function’s constraints. A lower resolution setting constructs the thumbnail more quickly, but at a loss of detail. +1. Movie. This uses the ui.Thumbnail API function to draw the BULC results rapidly in the viewer. This option offers you a control on the frame rate (in frames per second), and a checkbox affecting the drawing resolution. The high-resolution option uses the maximum resolution permitted given the function’s constraints. A lower resolution setting constructs the thumbnail more quickly, but at a loss of detail. 2. Filmstrip. This produces an image like the Movie option, but allows you to move on request through each image. -3. Mosaic. This draws every BULC result in the panel. Depending on the size of the stack of classifications, this could become quite a large set of images. -4. Zoom. This draws the final BULC classification at multiple scales, with the finest-scale image matching that shown in the Map window. +3. Mosaic. This draws every BULC result in the panel. Depending on the size of the stack of classifications, this could become quite a large set of images. +4. Zoom. This draws the final BULC classification at multiple scales, with the finest-scale image matching that shown in the Map window. -Question 2. Select the BULC option, then select the Movie tool to view the result, and choose a drawing speed and resolution. When viewing the full area, would you assess the additional LULC changes since 2016 as being minor, moderate, or major compared to the changes that occurred before 2016? Explain the reasoning for your assessment. +Question 2. Select the BULC option, then select the Movie tool to view the result, and choose a drawing speed and resolution. When viewing the full area, would you assess the additional LULC changes since 2016 as being minor, moderate, or major compared to the changes that occurred before 2016? Explain the reasoning for your assessment. ## Detailed LULC Inspection with BULC -BULC results can be viewed interactively, allowing you to view more detailed estimations of the LULC around the study area. We will zoom into a specific area where change did occur after 2016. To do that, turn on the Satellite view and zoom in. Watching the scale bar in the lower right of the Map panel, continue zooming until the scale bar says 5 km. Then, enter "-60.742, -9.844" in the Earth Engine search tool, located above the code. The text will be interpreted as a longitude/latitude value and will offer you a nearby coordinate, indicated with a value for the degrees West and the degrees South. Click that entry and Earth Engine will move to that location, while keeping at the specified zoom level. Let’s compare the BULC result in this sector against the image from Earth Engine’s satellite view that is underneath it (Fig. 4.8.4). +BULC results can be viewed interactively, allowing you to view more detailed estimations of the LULC around the study area. We will zoom into a specific area where change did occur after 2016. To do that, turn on the Satellite view and zoom in. Watching the scale bar in the lower right of the Map panel, continue zooming until the scale bar says 5 km. Then, enter "-60.742, -9.844" in the Earth Engine search tool, located above the code. The text will be interpreted as a longitude/latitude value and will offer you a nearby coordinate, indicated with a value for the degrees West and the degrees South. Click that entry and Earth Engine will move to that location, while keeping at the specified zoom level. Let’s compare the BULC result in this sector against the image from Earth Engine’s satellite view that is underneath it (Fig. 4.8.4). -![](F4/image51.png)  +![](F4/image51.png) -![](F4/image10.png) +![Fig. 4.8.4 Comparison of the final classification of the northern part of the study area to the satellite view](F4/image10.png) -Fig. 4.8.4 Comparison of the final classification of the northern part of the study area to the satellite view -BULC captured the changes between 2016 and 2021 with a classification series that suggests agricultural development (Fig. 4.8.4, left). Given the appearance of BULC’s 2021 classification, it suggests that the satellite backdrop at the time of this writing (Fig. 4.8.4, right) came from an earlier time period. +BULC captured the changes between 2016 and 2021 with a classification series that suggests agricultural development (Fig. 4.8.4, left). Given the appearance of BULC’s 2021 classification, it suggests that the satellite backdrop at the time of this writing (Fig. 4.8.4, right) came from an earlier time period. -Now, in the Results panel, select BULC, then Movie. Set your desired frame speed and resolution, then select Redraw Thumbnail. Then, zoom the main Map even closer to some agriculture that appears to have been established between 2016 and 2021. Redraw the thumbnail movie as needed to find an interesting set of pixels. +Now, in the Results panel, select BULC, then Movie. Set your desired frame speed and resolution, then select Redraw Thumbnail. Then, zoom the main Map even closer to some agriculture that appears to have been established between 2016 and 2021. Redraw the thumbnail movie as needed to find an interesting set of pixels. -With this finer-scale access to the results of BULC, you can select individual pixels to inspect. Move the horizontal divider downward to expose the Inspector tab and Console tab. Use the Inspector to click on several pixels to learn their history as expressed in the inputted Events and in BULC’s interpretation of the noise and signal in the Event series. In a chosen pixel, you might see output that looks like Fig. 4.8.5. It indicates a possible conversion in the Event time series after a few classifications of the pixel as Forest. This decreases the confidence that the pixel is still Forest (Fig. 4.8.5, lower panel), but not enough for the Active Agriculture class (class 3) to become the dominant probability. After the subsequent Event labels the pixel as Forest, the confidence (lower panel) recovers slightly, but not to its former level. The next Event classifies the pixel as Active Agriculture, confidently, by interpreting that second Active Agriculture classification, in a setting where change was already somewhat suspected after the first non-Forest classification. BULC’s label (middle panel) changes to be Active Agriculture at that point in the sequence. Subsequent Event classifications as Active Agriculture creates a growing confidence that its proper label at the end of the sequence was indeed Active Agriculture. +With this finer-scale access to the results of BULC, you can select individual pixels to inspect. Move the horizontal divider downward to expose the Inspector tab and Console tab. Use the Inspector to click on several pixels to learn their history as expressed in the inputted Events and in BULC’s interpretation of the noise and signal in the Event series. In a chosen pixel, you might see output that looks like Fig. 4.8.5. It indicates a possible conversion in the Event time series after a few classifications of the pixel as Forest. This decreases the confidence that the pixel is still Forest (Fig. 4.8.5, lower panel), but not enough for the Active Agriculture class (class 3) to become the dominant probability. After the subsequent Event labels the pixel as Forest, the confidence (lower panel) recovers slightly, but not to its former level. The next Event classifies the pixel as Active Agriculture, confidently, by interpreting that second Active Agriculture classification, in a setting where change was already somewhat suspected after the first non-Forest classification. BULC’s label (middle panel) changes to be Active Agriculture at that point in the sequence. Subsequent Event classifications as Active Agriculture creates a growing confidence that its proper label at the end of the sequence was indeed Active Agriculture. -![](F4/image23.png)![](F4/image72.png)![](F4/image58.png) +![Fig. 4.8.5 History for 2016-2020 for a pixel that appeared to have been newly cultivated during that period. (above): the input classifications, which s](F4/image23.png)![Fig. 4.8.5 History for 2016-2020 for a pixel that appeared to have been newly cultivated during that period. (above): the input classifications, which s](F4/image72.png)![Fig. 4.8.5 History for 2016-2020 for a pixel that appeared to have been newly cultivated during that period. (above): the input classifications, which s](F4/image58.png) -Fig. 4.8.5 History for 2016-2020 for a pixel that appeared to have been newly cultivated during that period. (above): the input classifications, which s -Question 3. Run the code again with the same data, but adjust the three levelers, then view the results presented in the Map window and the Results panel. How do each of the three parameters affect the behavior of BULC in its results? Use the thumbnail to assess your subjective satisfaction with the results, and use the Inspector to view the BULC behavior in individual pixels. Can you produce an optimal outcome for this given set of input classifications? +Question 3. Run the code again with the same data, but adjust the three levelers, then view the results presented in the Map window and the Results panel. How do each of the three parameters affect the behavior of BULC in its results? Use the thumbnail to assess your subjective satisfaction with the results, and use the Inspector to view the BULC behavior in individual pixels. Can you produce an optimal outcome for this given set of input classifications? ## Change Detection with BULC-D -What if we wanted to identify areas of likely change or stability without trying to identify the initial and final LULC class? BULC-D is an algorithm that estimates, at each time step, the probability of noteworthy change. The example below uses the Normalized Burn Ratio (NBR) as a gauge: BULC-D assesses whether the ratio has meaningfully increased, decreased, or remained the same. It is then the choice of the analyst to decide how to treat these assessed probabilities of stability and change. +What if we wanted to identify areas of likely change or stability without trying to identify the initial and final LULC class? BULC-D is an algorithm that estimates, at each time step, the probability of noteworthy change. The example below uses the Normalized Burn Ratio (NBR) as a gauge: BULC-D assesses whether the ratio has meaningfully increased, decreased, or remained the same. It is then the choice of the analyst to decide how to treat these assessed probabilities of stability and change. BULC-D involves determining an expectation for an index across a user-specified time period and then comparing new values against that estimation. Using Bayesian logic, BULC-D then asks which of three hypotheses is most likely, given evidence from the new values to date from that index. The hypotheses are simple: Either the value has decreased meaningfully, or it has increased meaningfully, or it has not changed substantially compared to the previously established expectation. The details of the workings of BULC-D are beyond the scope of this exercise, but we provide it as a tool for exploration. BULC-D’s basic framework is the following: @@ -2582,10 +2619,10 @@ BULC-D involves determining an expectation for an index across a user-specified It is worth noting that BULC-D does not label the change with a LULC category; rather, it trains itself to distinguish likely LULC change from expected variability. In this way, BULC-D can be thought of as a “sieve” through which you are able to identify locations of possible change, isolated from likely background noise. In the BULC-D stage, the likeliness of change is identified across the landscape; in a separate second stage, the meaning of those changes and any changes to LULC classes are identified. We will explore the workings of BULC-D using its GUI. -::: {.callout-note} -Code Checkpoint F48c. The book’s repository contains information about accessing that interface. +:::{.callout-note} +Code Checkpoint F48c. The book’s repository contains information about accessing that interface. ::: -After you have run the script to initialize the interface, BULC-D’s interface requires a few parameters to be set. For this run of BULC-D, we will set the parameters to the following: +After you have run the script to initialize the interface, BULC-D’s interface requires a few parameters to be set. For this run of BULC-D, we will set the parameters to the following: * Expectation years: 2020 * Target year: 2021 @@ -2595,39 +2632,37 @@ After you have run the script to initialize the interface, BULC-D’s interface Run BULC-D for this area. As a reminder, you should first zoom in enough that the scale bar reads “5 km” or finer. Then, search for the location "-60.7624, -9.8542". When you run BULC-D, a result like Fig. F4.8.6 is shown for the layer of probabilities. -![](F4/image48.png)  +![](F4/image48.png) -Fig. 4.8.6 Result for BULC-D for the Roosevelt River area, depicting estimated probability of change and stability for 2021 +Fig. 4.8.6 Result for BULC-D for the Roosevelt River area, depicting estimated probability of change and stability for 2021 -The BULC-D image (Fig. F4.8.6) shows each pixel as a continuous three-value vector along a continuous range; the three values sum to 1. For example, a vector with values of [0.85, 0.10, 0.05] would represent an area estimated with high confidence according to BULC-D to have experienced a sustained drop in NBR in the target period compared to the values set by the expectation data. In that pixel, the combination of three colors would produce a value that is richly red. You can see Chap. F1.1 for more information on drawing bands of information to the screen using the red-green-blue additive color model in Earth Engine. +The BULC-D image (Fig. F4.8.6) shows each pixel as a continuous three-value vector along a continuous range; the three values sum to 1. For example, a vector with values of [0.85, 0.10, 0.05] would represent an area estimated with high confidence according to BULC-D to have experienced a sustained drop in NBR in the target period compared to the values set by the expectation data. In that pixel, the combination of three colors would produce a value that is richly red. You can see Chap. F1.1 for more information on drawing bands of information to the screen using the red-green-blue additive color model in Earth Engine. -Each pixel experiences its own NBR history in both the expectation period and the target year. Next, we will highlight the history of three nearby areas: one, marked with a red balloon in your interface, that BULC assessed as having experienced a persistent drop in NBR; a second in green assessed to not have changed, and a third in blue assessed to have witnessed a persistent NBR increase.  +Each pixel experiences its own NBR history in both the expectation period and the target year. Next, we will highlight the history of three nearby areas: one, marked with a red balloon in your interface, that BULC assessed as having experienced a persistent drop in NBR; a second in green assessed to not have changed, and a third in blue assessed to have witnessed a persistent NBR increase. -Figure F4.8.7 shows the NBR history for the red balloon in the southern part of the study area in Fig. F4.8.4. If you click on that pixel or one like it, you can see that, whereas the values were quite stable throughout the growing season for the years used to create the pixel’s expectation, they were persistently lower in the target year. This is flagged as a likely meaningful drop in the NBR by BULC-D, for consideration by the analyst. +Figure F4.8.7 shows the NBR history for the red balloon in the southern part of the study area in Fig. F4.8.4. If you click on that pixel or one like it, you can see that, whereas the values were quite stable throughout the growing season for the years used to create the pixel’s expectation, they were persistently lower in the target year. This is flagged as a likely meaningful drop in the NBR by BULC-D, for consideration by the analyst. -![](F4/image79.png)                                                                                        +![](F4/image79.png) -Fig. 4.8.7 NBR history for a pixel with an apparent drop in NBR in the target year (below) as compared to the expectation years (above). Pixel is colored a shade of red in Fig. 4.8.6. +Fig. 4.8.7 NBR history for a pixel with an apparent drop in NBR in the target year (below) as compared to the expectation years (above). Pixel is colored a shade of red in Fig. 4.8.6. -Figure F4.8.8 shows the NBR history for the blue balloon in the southern part of the study area in Fig. F4.8.4. For that pixel, while the values were quite stable throughout the growing season for the years used to create the pixel’s expectation, they were persistently higher in the target year.  +Figure F4.8.8 shows the NBR history for the blue balloon in the southern part of the study area in Fig. F4.8.4. For that pixel, while the values were quite stable throughout the growing season for the years used to create the pixel’s expectation, they were persistently higher in the target year. -Question 4. Experiment with turning off one of the satellite sensor data sources used to create the expectation collection. For example, do you get the same results if the Sentinel-2 data stream is not used, or is the outcome different. You might make screen captures of the results to compare with Fig. 4.8.4. How strongly does each satellite stream affect the outcome of the estimate?  Do differences in the resulting estimate vary across the study area? +Question 4. Experiment with turning off one of the satellite sensor data sources used to create the expectation collection. For example, do you get the same results if the Sentinel-2 data stream is not used, or is the outcome different. You might make screen captures of the results to compare with Fig. 4.8.4. How strongly does each satellite stream affect the outcome of the estimate? Do differences in the resulting estimate vary across the study area? -![](F4/image52.png) +![Fig. 4.8.8 NBR history for a pixel with an apparent increase in NBR in the target year (below) as compared to the expectation years (above). Pixel is colored a shade of blue in Fig. 4.8.6.](F4/image52.png) -Fig. 4.8.8 NBR history for a pixel with an apparent increase in NBR in the target year (below) as compared to the expectation years (above). Pixel is colored a shade of blue in Fig. 4.8.6. Figure F4.8.8 also shows that, for that pixel, the fit of values for the years used to build the expectation showed a sine wave (shown in blue), but with a fit that was not very strong. When data for the target year was assembled (Fig. F4.8.8, bottom), the values were persistently above expectation throughout the growing season. Note that this pixel was identified as being different in the target year as compared to earlier years, which does not rule out the possibility that the LULC of the area was changed (for example, from Forest to Agriculture) during the years used to build the expectation collection. BULC-D is intended to be run steadily over a long period of time, with the changes marked as they occur, after which point the expectation would be recalculated. -![](F4/image56.png) +![Fig. 4.8.9 NBR history for a pixel with no apparent increase or decrease in NBR in the target year (below) as compared to the expectation years (above). Pixel is colored a shade of green in Fig. 4.8.6.](F4/image56.png) -Fig. 4.8.9 NBR history for a pixel with no apparent increase or decrease in NBR in the target year (below) as compared to the expectation years (above). Pixel is colored a shade of green in Fig. 4.8.6. Fig. F4.8.9 shows the NBR history for the green balloon in the southern part of the study area in Fig. F4.8.4. For that pixel, the values in the expectation collection formed a sine wave, and the values in the target collection deviated only slightly from the expectation during the target year. ## Change Detection with BULC and Dynamic World -Recent advances in neural networks have made it easier to develop consistent models of LULC characteristics using satellite data. The Dynamic World project (Brown et al. 2022) applies a neural network, trained on a very large number of images, to each new Sentinel-2 image soon after it arrives. The result is a near-real-time classification interpreting the LULC of Earth’s surface, kept continually up to date with new imagery. +Recent advances in neural networks have made it easier to develop consistent models of LULC characteristics using satellite data. The Dynamic World project (Brown et al. 2022) applies a neural network, trained on a very large number of images, to each new Sentinel-2 image soon after it arrives. The result is a near-real-time classification interpreting the LULC of Earth’s surface, kept continually up to date with new imagery. What to do with the inevitable inconsistencies in a pixel’s stated LULC class through time? For a given pixel on a given image, its assigned class label is chosen by the Dynamic World algorithm as the maximum class probability given the band values on that day. Individual class probabilities are given as part of the dataset and could be used to better interpret a pixel’s condition and perhaps its history. Future work with BULC will involve incorporating these probabilities into BULC’s probability-based structure. For this tutorial, we will explore the consistency of the assigned labels in this same Roosevelt River area as a way to illustrate BULC’s potential for minimizing noise in this vast and growing dataset. @@ -2635,42 +2670,39 @@ What to do with the inevitable inconsistencies in a pixel’s stated LULC class Code Checkpoint A48d. The book’s repository contains a script to use to begin this section. You will need to load the linked script and run it to begin. -After running the linked script, the BULC interface will initialize. Select Dynamic World from the dropdown menu where you earlier selected Image Collection. When you do, the interface opens several new fields to complete. BULC will need to know where you are interested in working with Dynamic World, since it could be anywhere on Earth. To specify the location, the interface field expects a nested list of lists of lists, which is modeled after the structure used inside the constructor ee.Geometry.Polygon. (When using drawing tools or specifying study areas using coordinates, you may have noticed this structure.) Enter the following nested list in the text field near the Dynamic World option, without enclosing it in quotes: +After running the linked script, the BULC interface will initialize. Select Dynamic World from the dropdown menu where you earlier selected Image Collection. When you do, the interface opens several new fields to complete. BULC will need to know where you are interested in working with Dynamic World, since it could be anywhere on Earth. To specify the location, the interface field expects a nested list of lists of lists, which is modeled after the structure used inside the constructor ee.Geometry.Polygon. (When using drawing tools or specifying study areas using coordinates, you may have noticed this structure.) Enter the following nested list in the text field near the Dynamic World option, without enclosing it in quotes: [[[-61.155, -10.559], [-60.285, -10.559], [-60.285, -9.436], [-61.155, -9.436]]] -Next, BULC will need to know which years of Dynamic World you are interested in. For this exercise, select 2021. Then, BULC will ask for the Julian days of the year that you are interested in. For this exercise, enter 150 for the start day and 300 for the end day. Because you selected Dynamic World for analysis in BULC, the interface defaults to offering the number 9 for the number of classes in Events and for the number of classes to track. This number represents the full set of classes in the Dynamic World classification scheme. You can leave other required settings shown in green with their default values. For the Color Output Palette, enter the following palette without quotes. This will render results in the Dynamic World default colors. +Next, BULC will need to know which years of Dynamic World you are interested in. For this exercise, select 2021. Then, BULC will ask for the Julian days of the year that you are interested in. For this exercise, enter 150 for the start day and 300 for the end day. Because you selected Dynamic World for analysis in BULC, the interface defaults to offering the number 9 for the number of classes in Events and for the number of classes to track. This number represents the full set of classes in the Dynamic World classification scheme. You can leave other required settings shown in green with their default values. For the Color Output Palette, enter the following palette without quotes. This will render results in the Dynamic World default colors. ['419BDF', '397D49', '88B053', '7A87C6', 'E49635', 'DFC35A', 'C4281B', 'A59B8F', 'B39FE1'] -When you have finished, select Apply Parameters at the bottom of the input panel. When it runs, BULC subsets the Dynamic World dataset to clip out according to the dates and location, identifying images from more than 40 distinct dates. The area covers two of the tiles in which Dynamic World classifications are partitioned to be served, so BULC receives more than 90 classifications. When BULC finishes its run, the Map panel will look like Fig. F4.8.10, BULC’s estimate of the final state of the landscape at the end of the classification sequence. +When you have finished, select Apply Parameters at the bottom of the input panel. When it runs, BULC subsets the Dynamic World dataset to clip out according to the dates and location, identifying images from more than 40 distinct dates. The area covers two of the tiles in which Dynamic World classifications are partitioned to be served, so BULC receives more than 90 classifications. When BULC finishes its run, the Map panel will look like Fig. F4.8.10, BULC’s estimate of the final state of the landscape at the end of the classification sequence. -![](F4/image37.png)![](F4/image41.png) +![Fig. F4.8.10 BULC classification using default settings for Roosevelt River area for late 2021](F4/image37.png)![Fig. F4.8.10 BULC classification using default settings for Roosevelt River area for late 2021](F4/image41.png) -Fig. F4.8.10 BULC classification using default settings for Roosevelt River area for late 2021 -Let’s explore the suite of information returned by BULC about this time period in Dynamic World. Enter “Muiraquitã” in the search bar and view the results around that area to be able to see the changing LULC classifications within farm fields. Then, begin to inspect the results by viewing a Movie of the Events, with a data frame rate of 6 frames per second. Because the study area spans multiple Dynamic World tiles, you will find that many Event frames are black, meaning that there was no data in your sector on that particular image. Because of this, and also perhaps because of the very aggressive cloud masking built into Dynamic World, viewing Events (which, as a reminder, are the individual classified images directly from Dynamic World) can be a very challenging way to look for change and stability. BULC’s goal is to sift through those classifications to produce a time series that reflects, according to its estimation, the most likely LULC value at each time step. View the Movie of the BULC results and ask yourself whether each class is equally well replicated across the set of classifications. A still from midway through the Movie sequence of the BULC results can be seen in Fig. F4.8.11. +Let’s explore the suite of information returned by BULC about this time period in Dynamic World. Enter “Muiraquitã” in the search bar and view the results around that area to be able to see the changing LULC classifications within farm fields. Then, begin to inspect the results by viewing a Movie of the Events, with a data frame rate of 6 frames per second. Because the study area spans multiple Dynamic World tiles, you will find that many Event frames are black, meaning that there was no data in your sector on that particular image. Because of this, and also perhaps because of the very aggressive cloud masking built into Dynamic World, viewing Events (which, as a reminder, are the individual classified images directly from Dynamic World) can be a very challenging way to look for change and stability. BULC’s goal is to sift through those classifications to produce a time series that reflects, according to its estimation, the most likely LULC value at each time step. View the Movie of the BULC results and ask yourself whether each class is equally well replicated across the set of classifications. A still from midway through the Movie sequence of the BULC results can be seen in Fig. F4.8.11. -![](F4/image13.png) +![Fig. F4.8.11 Still frame (right image) from the animation of BULC’s adjusted estimate of LULC through time near Muiraquitã](F4/image13.png) -Fig. F4.8.11 Still frame (right image) from the animation of BULC’s adjusted estimate of LULC through time near Muiraquitã As BULC uses the classification inputs to estimate the state of the LULC at each time step, it also tracks its confidence in those estimates. This is shown in several ways in the interface. -* You can view a Movie of BULC’s confidence through time as it reacts to the consistency or variability of the class identified in each pixel by Dynamic World. View that movie now over this area to see the evolution of BULC’s confidence through time of the class of each pixel. A still frame from this movie can be seen in Fig. F4.8.12. The frame and animation indicate that BULC’s confidence is lowest in pixels where the estimate flips between similar categories, such as Grass and Shrub & Scrub. It also is low at the edges of land covers, even where the covers (such as Forest and Water) are easy to discern from each other.   -* You can inspect the final confidence estimate from BULC, which is shown as a grayscale image in the set of Map layers in the left lower panel. That single layer synthesizes how, across many Dynamic World classifications, the confidence in certain LULC classes and locations is ultimately more stable than in others. For example, generally speaking, the Forest class is classified consistently across this assemblage of Dynamic World images. Agriculture fields are less consistently classified as a single class, as evidenced by their relatively low confidence. -* Another way of viewing BULC’s confidence is through the Inspector tab. You can click on individual pixels to view their values in the Event time series and in the BULC time series, and see BULC’s corresponding confidence value changing through time in response to the relative stability of each pixel’s classification. +* You can view a Movie of BULC’s confidence through time as it reacts to the consistency or variability of the class identified in each pixel by Dynamic World. View that movie now over this area to see the evolution of BULC’s confidence through time of the class of each pixel. A still frame from this movie can be seen in Fig. F4.8.12. The frame and animation indicate that BULC’s confidence is lowest in pixels where the estimate flips between similar categories, such as Grass and Shrub & Scrub. It also is low at the edges of land covers, even where the covers (such as Forest and Water) are easy to discern from each other. +* You can inspect the final confidence estimate from BULC, which is shown as a grayscale image in the set of Map layers in the left lower panel. That single layer synthesizes how, across many Dynamic World classifications, the confidence in certain LULC classes and locations is ultimately more stable than in others. For example, generally speaking, the Forest class is classified consistently across this assemblage of Dynamic World images. Agriculture fields are less consistently classified as a single class, as evidenced by their relatively low confidence. +* Another way of viewing BULC’s confidence is through the Inspector tab. You can click on individual pixels to view their values in the Event time series and in the BULC time series, and see BULC’s corresponding confidence value changing through time in response to the relative stability of each pixel’s classification. * Another way to view BULC’s confidence estimation is as a hillshade enhancement of the final BULC classification. If you select the Probability Hillshade in the set of Map layers, it shows the final BULC classification as a textured surface, in which you can see where lower-confidence pixels are classified. -![](F4/image40.png) +![Fig. F4.8.12 Still frame from the animation of changing confidence through time, near Muiraquitã.](F4/image40.png) -Fig. F4.8.12 Still frame from the animation of changing confidence through time, near Muiraquitã. ### 5.2. Using BULC To Visualize Uncertainty of Dynamic World in Simplified Categories -In the previous section, you may have noticed that there are two main types of uncertainty in BULC’s assessment of long-term classification confidence. One type is due to spatial uncertainty at the edge of two relatively distinct phenomena, like the River/Forest boundary visible in Fig. F4.8.12. These are shown in dark tones in the confidence images, and emphasized in the Probability Hillshade. The other type of uncertainty is due to some cause of labeling uncertainty, due either (1) to the similarity of the classes, or (2) to persistent difficulty in distinguishing two distinct classes that are meaningfully different but spectrally similar. An example of uncertainty due to similar labels is distinguishing flooded and non-flooded wetlands in classifications that contain both those categories. An example of difficulty distinguishing distinct but spectrally similar classes might be distinguishing a parking lot from a body of water. +In the previous section, you may have noticed that there are two main types of uncertainty in BULC’s assessment of long-term classification confidence. One type is due to spatial uncertainty at the edge of two relatively distinct phenomena, like the River/Forest boundary visible in Fig. F4.8.12. These are shown in dark tones in the confidence images, and emphasized in the Probability Hillshade. The other type of uncertainty is due to some cause of labeling uncertainty, due either (1) to the similarity of the classes, or (2) to persistent difficulty in distinguishing two distinct classes that are meaningfully different but spectrally similar. An example of uncertainty due to similar labels is distinguishing flooded and non-flooded wetlands in classifications that contain both those categories. An example of difficulty distinguishing distinct but spectrally similar classes might be distinguishing a parking lot from a body of water. -BULC allows you to remap the classifications it is given as input, compressing categories as a way to minimize uncertainty due to similarity among classes. In the setting of Dynamic World in this study area, we notice that several classes are functionally similar for the purposes of detecting new deforestation: Farm fields and pastures are variously labeled on any given Dynamic World classification as Grass, Flooded Vegetation, Crops, Shrub & Scrub, Built, or Bare Ground. What if we wanted to combine these categories to be similar to the distinctions of the classified Events from this lab’s Sect. 1? The classes in that section were Forest, Water, and Active Agriculture. To remap the Dynamic World classification, continue with the same run as in Sect. 5.1. Near where you specified the location for clipping Dynamic World, there are two fields for remapping. Select the Remap checkbox and in the “from” field, enter (without quotes): +BULC allows you to remap the classifications it is given as input, compressing categories as a way to minimize uncertainty due to similarity among classes. In the setting of Dynamic World in this study area, we notice that several classes are functionally similar for the purposes of detecting new deforestation: Farm fields and pastures are variously labeled on any given Dynamic World classification as Grass, Flooded Vegetation, Crops, Shrub & Scrub, Built, or Bare Ground. What if we wanted to combine these categories to be similar to the distinctions of the classified Events from this lab’s Sect. 1? The classes in that section were Forest, Water, and Active Agriculture. To remap the Dynamic World classification, continue with the same run as in Sect. 5.1. Near where you specified the location for clipping Dynamic World, there are two fields for remapping. Select the Remap checkbox and in the “from” field, enter (without quotes): 0,1,2,3,4,5,6,7,8 @@ -2682,13 +2714,12 @@ This directs BULC to create a three-class remap of each Dynamic World image. Nex ['green', 'blue', 'yellow'] -Before continuing, think for a moment about how many classes you have now. From BULC’s perspective, the Dynamic World events will have 3 classes and you will be tracking 3 classes. Set both the Number of Classes in Events and Number of Classes to Track to 3. Then click Apply Parameters to send this new run to BULC. +Before continuing, think for a moment about how many classes you have now. From BULC’s perspective, the Dynamic World events will have 3 classes and you will be tracking 3 classes. Set both the Number of Classes in Events and Number of Classes to Track to 3. Then click Apply Parameters to send this new run to BULC. -The confidence image shown in the main Map panel is instructive (Fig. 4.8.13). Using data from 2020, 2021, and 2022, It indicates that much of the uncertainty among the original Dynamic World classifications was in distinguishing labels within agricultural fields. When that uncertainty is removed by combining classes, the BULC result indicates that a substantial part of the remaining uncertainty is at the edges of distinct covers. For example, in the south-central and southern part of the frame, much of the uncertainty among classifications in the original Dynamic World classifications was due to distinction among the highly similar, easily confused classes. Much of what remained (right) after remapping (right) formed outlines of the river and the edges between farmland and forest: a graphic depiction of the “spatial uncertainty” discussed earlier. Yet not all of the uncertainty was spatial; the thicker, darker areas of uncertainty even after remapping (right, at the extreme eastern edge for example) indicates a more fundamental disagreement in the classifications. In those pixels, even when the Agriculture-like classes were compressed, there was still considerable uncertainty (likely between Forest and Active Agriculture) in the true state of these areas. These might be of further interest: were they places newly deforested in 2020-2022? Were they abandoned fields regrowing?  Were they degraded at some point? The mapping of uncertainty may hold promise for a better understanding of uncertainty as it is encountered in real classifications, thanks to Dynamic World. +The confidence image shown in the main Map panel is instructive (Fig. 4.8.13). Using data from 2020, 2021, and 2022, It indicates that much of the uncertainty among the original Dynamic World classifications was in distinguishing labels within agricultural fields. When that uncertainty is removed by combining classes, the BULC result indicates that a substantial part of the remaining uncertainty is at the edges of distinct covers. For example, in the south-central and southern part of the frame, much of the uncertainty among classifications in the original Dynamic World classifications was due to distinction among the highly similar, easily confused classes. Much of what remained (right) after remapping (right) formed outlines of the river and the edges between farmland and forest: a graphic depiction of the “spatial uncertainty” discussed earlier. Yet not all of the uncertainty was spatial; the thicker, darker areas of uncertainty even after remapping (right, at the extreme eastern edge for example) indicates a more fundamental disagreement in the classifications. In those pixels, even when the Agriculture-like classes were compressed, there was still considerable uncertainty (likely between Forest and Active Agriculture) in the true state of these areas. These might be of further interest: were they places newly deforested in 2020-2022? Were they abandoned fields regrowing? Were they degraded at some point? The mapping of uncertainty may hold promise for a better understanding of uncertainty as it is encountered in real classifications, thanks to Dynamic World. -![](F4/image47.png)![](F4/image89.png) +![Fig. F4.8.13 Final confidence layer from the run with (left) and without (right) remapping to combine similar LULC classes to distinguish Forest, Water, and Active Agriculture near -60.696W, -9.826S](F4/image47.png)![Fig. F4.8.13 Final confidence layer from the run with (left) and without (right) remapping to combine similar LULC classes to distinguish Forest, Water, and Active Agriculture near -60.696W, -9.826S](F4/image89.png) -Fig. F4.8.13 Final confidence layer from the run with (left) and without (right) remapping to combine similar LULC classes to distinguish Forest, Water, and Active Agriculture near -60.696W, -9.826S Given the tools and approaches presented in this lab, you should now be able to import your own classifications for BULC (Sects. 1–3), detect changes in sets of raw imagery (Sect. 4), or use Dynamic World’s pre-created classifications (Sect. 5). The following exercises explore this potential. @@ -2696,14 +2727,14 @@ Given the tools and approaches presented in this lab, you should now be able to Assignment 1. For a given set of classifications as inputs, BULC uses three parameters that specify how strongly to trust the initial classification, how heavily to weigh the evidence of each classification, and how to adjust the confidence at the end of each time step. For this exercise, adjust the values of these three parameters to explore the strength of the effect they can have on the BULC results. -Assignment 2. The BULC-D framework produces a continuous three-value vector of the probability of change at each pixel. This variability accounts for the mottled look of the figures when those probabilities are viewed across space. Use the Inspector tool or the interface to explore the final estimated probabilities, both numerically and as represented by different colors of pixels in the given example. Compare and contrast the mean NBR values from the earlier and later years, which are drawn in the Layer list. Then answer the following questions: +Assignment 2. The BULC-D framework produces a continuous three-value vector of the probability of change at each pixel. This variability accounts for the mottled look of the figures when those probabilities are viewed across space. Use the Inspector tool or the interface to explore the final estimated probabilities, both numerically and as represented by different colors of pixels in the given example. Compare and contrast the mean NBR values from the earlier and later years, which are drawn in the Layer list. Then answer the following questions: -1. In general, how well does BULC-D appear to be identifying locations of likely change? +1. In general, how well does BULC-D appear to be identifying locations of likely change? 2. Does one type of change (decrease, increase, no change) appear to be mapped better than the others? If so, why do you think this is? Assignment 3. The BULC-D example used here was for 2021. Run it for 2022 or later at this location. How well do results from adjacent years complement each other? -Assignment 4. Run BULC-D in a different area for a year of interest of your choosing. How do you like the results? +Assignment 4. Run BULC-D in a different area for a year of interest of your choosing. How do you like the results? Assignment 5. Describe how you might use BULC-D as a filter for distinguishing meaningful change from noise. In your answer, you can consider using BULC-D before or after BULC or some other time-series algorithm, like CCDC or LandTrendr. @@ -2715,19 +2746,19 @@ Assignment 6. Analyze stability and change with Dynamic World for other parts of Location of a summer 2020 fire -2. Addis Ababa, Ethiopia:  [[[38.79, 9.00], [38.79, 8.99], [38.81, 8.99], [38.81, 9.00]]] +2. Addis Ababa, Ethiopia: [[[38.79, 9.00], [38.79, 8.99], [38.81, 8.99], [38.81, 9.00]]] 3. Calacalí, Ecuador: [[[-78.537, 0.017], [-78.537, -0.047], [-78.463, -0.047], [-78.463, 0.017]]] 4. Irpin, Ukraine: [[[30.22, 50.58], [30.22, 50.525], [30.346, 50.525], [30.346, 50.58]]] -5. A different location of your own choosing. To do this, use the Earth Engine drawing tools to draw a rectangle somewhere on Earth. Then, at the top of the Import section, you will see an icon that looks like a sheet of paper. Click that icon and look for the polygon specification for the rectangle you drew. Paste that into the location field for the Dynamic World interface. +5. A different location of your own choosing. To do this, use the Earth Engine drawing tools to draw a rectangle somewhere on Earth. Then, at the top of the Import section, you will see an icon that looks like a sheet of paper. Click that icon and look for the polygon specification for the rectangle you drew. Paste that into the location field for the Dynamic World interface. ## Conclusion {.unnumbered} In this lab, you have viewed several related but distinct ways to use Bayesian statistics to identify locations of LULC change in complex landscapes. While they are standalone algorithms, they are each intended to provide a perspective either on the likelihood of change (BULC-D) or of extracting signal from noisy classifications (BULC). You can consider using them especially when you have pixels that, despite your best efforts, periodically flip back and forth between similar but different classes. BULC can help ignore noise, and BULC-D can help reveal whether this year’s signal has precedent in past years. -To learn more about the BULC algorithm, you can view this interactive probability illustration tool by a link found in script F48s1 - Supplemental in the book's repository. In the future, after you have learned how to use the logic of BULC, you might prefer to work with the JavaScript code version. To do that, you can find a tutorial at the website of the authors. +To learn more about the BULC algorithm, you can view this interactive probability illustration tool by a link found in script F48s1 - Supplemental in the book's repository. In the future, after you have learned how to use the logic of BULC, you might prefer to work with the JavaScript code version. To do that, you can find a tutorial at the website of the authors. ## References {.unnumbered} @@ -2745,13 +2776,13 @@ Millard C (2006) The River of Doubt: Theodore Roosevelt’s Darkest Journey. Anc -# Exploring Lagged Effects in Time Series  +# Exploring Lagged Effects in Time Series -::: {.callout-tip} +:::{.callout-tip} # Chapter Information ## Author {.unlisted .unnumbered} @@ -2765,12 +2796,12 @@ Andréa Puzzi Nicolau, Karen Dyson, David Saah, Nicholas Clinton ## Overview {.unlisted .unnumbered} -In this chapter, we will introduce lagged effects to build on previous work in modeling time-series data. Time-lagged effects occur when an event at one point in time impacts dependent variables at a later point in time. You will be introduced to concepts of autocovariance and autocorrelation, cross-covariance and cross-correlation, and auto-regressive models. At the end of this chapter, you will be able to examine how variables relate to one another across time, and to fit time series models that take into account lagged events.  +In this chapter, we will introduce lagged effects to build on previous work in modeling time-series data. Time-lagged effects occur when an event at one point in time impacts dependent variables at a later point in time. You will be introduced to concepts of autocovariance and autocorrelation, cross-covariance and cross-correlation, and auto-regressive models. At the end of this chapter, you will be able to examine how variables relate to one another across time, and to fit time series models that take into account lagged events. ## Learning Outcomes {.unlisted .unnumbered} -* Using the ee.Join function to create time-lagged collections. +* Using the ee.Join function to create time-lagged collections. * Calculating autocovariance and autocorrelation. * Calculating cross-covariance and cross-correlation. * Fitting auto-regressive models. @@ -2780,12 +2811,12 @@ In this chapter, we will introduce lagged effects to build on previous work in m * Import images and image collections, filter, and visualize (Part F1). * Perform basic image analysis: select bands, compute indices, create masks, classify images (Part F2). -* ​​Create a graph using ui.Chart (Chap. F1.3). -* Write a function and map it over an ImageCollection (Chap. F4.0). +* ​​Create a graph using ui.Chart (Chap. F1.3). +* Write a function and map it over an ImageCollection (Chap. F4.0). * Mask cloud, cloud shadow, snow/ice, and other undesired pixels (Chap. F4.3). -* Fit linear and nonlinear functions with regression in an ImageCollection time series (Chap. F4.6). +* Fit linear and nonlinear functions with regression in an ImageCollection time series (Chap. F4.6). -::: +::: ## Introduction {.unlisted .unnumbered} @@ -2794,308 +2825,327 @@ While fitting functions to time series allows you to account for seasonality in In this chapter, we introduce lagged effects into our previous discussions on interpreting time-series data (Chaps. F4.6 and F4.7). Being able to integrate lagged effects into our time-series models allows us to address many important questions. For example, streamflow can be accurately modeled by taking into account previous streamflow, rainfall, and soil moisture; this improved understanding helps predict and mitigate the impacts of drought and flood events made more likely by climate change (Sazib et al. 2020). As another example, time-series lag analysis was able to determine that decreased rainfall was associated with increases in livestock disease outbreaks one year later in India (Karthikeyan et al. 2021). -## Autocovariance and Autocorrelation +## Autocovariance and Autocorrelation -Before we dive into autocovariance and autocorrelation, let’s set up an area of interest and dataset that we can use to illustrate these concepts. We will work with a detrended time series (as seen in Chap. F4.6) based on the USGS Landsat 8 Level 2, Collection 2, Tier 1 image collection. Copy and paste the code below to filter the Landsat 8 collection to a point of interest over California and specific dates, and apply the pre-processing function—to mask clouds (as seen in Chap. F4.3) and to scale and add variables of interest (as seen in Chap. F4.6). +Before we dive into autocovariance and autocorrelation, let’s set up an area of interest and dataset that we can use to illustrate these concepts. We will work with a detrended time series (as seen in Chap. F4.6) based on the USGS Landsat 8 Level 2, Collection 2, Tier 1 image collection. Copy and paste the code below to filter the Landsat 8 collection to a point of interest over California and specific dates, and apply the pre-processing function—to mask clouds (as seen in Chap. F4.3) and to scale and add variables of interest (as seen in Chap. F4.6). +```js // Define function to mask clouds, scale, and add variables // (NDVI, time and a constant) to Landsat 8 imagery. -function maskScaleAndAddVariable(image) {   // Bit 0 - Fill   // Bit 1 - Dilated Cloud   // Bit 2 - Cirrus   // Bit 3 - Cloud   // Bit 4 - Cloud Shadow   var qaMask = image.select('QA_PIXEL').bitwiseAnd(parseInt('11111',       2)).eq(0);   var saturationMask = image.select('QA_RADSAT').eq(0);   // Apply the scaling factors to the appropriate bands.   var opticalBands = image.select('SR_B.').multiply(0.0000275).add(-       0.2);   var thermalBands = image.select('ST_B.*').multiply(0.00341802) -       .add(149.0);   // Replace the original bands with the scaled ones and apply the masks.   var img = image.addBands(opticalBands, null, true) -       .addBands(thermalBands, null, true) -       .updateMask(qaMask) -       .updateMask(saturationMask);   var imgScaled = image.addBands(img, null, true);   // Now we start to add variables of interest.   // Compute time in fractional years since the epoch.   var date = ee.Date(image.get('system:time_start'));   var years = date.difference(ee.Date('1970-01-01'), 'year');   var timeRadians = ee.Image(years.multiply(2 * Math.PI)); -   // Return the image with the added bands. -   return imgScaled -       // Add an NDVI band. -       .addBands(imgScaled.normalizedDifference(['SR_B5', 'SR_B4']) -           .rename('NDVI'))       // Add a time band.       .addBands(timeRadians.rename('t')) -       .float()       // Add a constant band.       .addBands(ee.Image.constant(1)); +function maskScaleAndAddVariable(image) { // Bit 0 - Fill // Bit 1 - Dilated Cloud // Bit 2 - Cirrus // Bit 3 - Cloud // Bit 4 - Cloud Shadow var qaMask = image.select('QA_PIXEL').bitwiseAnd(parseInt('11111', 2)).eq(0); var saturationMask = image.select('QA_RADSAT').eq(0); // Apply the scaling factors to the appropriate bands. var opticalBands = image.select('SR_B.').multiply(0.0000275).add(- 0.2); var thermalBands = image.select('ST_B.*').multiply(0.00341802) + .add(149.0); // Replace the original bands with the scaled ones and apply the masks. var img = image.addBands(opticalBands, null, true) + .addBands(thermalBands, null, true) + .updateMask(qaMask) + .updateMask(saturationMask); var imgScaled = image.addBands(img, null, true); // Now we start to add variables of interest. // Compute time in fractional years since the epoch. var date = ee.Date(image.get('system:time_start')); var years = date.difference(ee.Date('1970-01-01'), 'year'); var timeRadians = ee.Image(years.multiply(2 * Math.PI)); + // Return the image with the added bands. + return imgScaled + // Add an NDVI band. + .addBands(imgScaled.normalizedDifference(['SR_B5', 'SR_B4']) + .rename('NDVI')) // Add a time band. .addBands(timeRadians.rename('t')) + .float() // Add a constant band. .addBands(ee.Image.constant(1)); } // Import region of interest. Area over California. -var roi = ee.Geometry.Polygon([ -   [-119.44617458417066,35.92639730653253], -   [-119.07675930096754,35.92639730653253], -   [-119.07675930096754,36.201704711823844], -   [-119.44617458417066,36.201704711823844], -   [-119.44617458417066,35.92639730653253] +var roi = ee.Geometry.Polygon([ + [-119.44617458417066,35.92639730653253], + [-119.07675930096754,35.92639730653253], + [-119.07675930096754,36.201704711823844], + [-119.44617458417066,36.201704711823844], + [-119.44617458417066,35.92639730653253] ]); // Import the USGS Landsat 8 Level 2, Collection 2, Tier 1 collection, // filter, mask clouds, scale, and add variables. -var landsat8sr = ee.ImageCollection('LANDSAT/LC08/C02/T1_L2') -   .filterBounds(roi) -   .filterDate('2013-01-01', '2018-01-01') -   .map(maskScaleAndAddVariable); +var landsat8sr = ee.ImageCollection('LANDSAT/LC08/C02/T1_L2') + .filterBounds(roi) + .filterDate('2013-01-01', '2018-01-01') + .map(maskScaleAndAddVariable); // Set map center. Map.centerObject(roi, 10); -Next, copy and paste the code below to estimate the linear trend using the linearRegression reducer, and remove that linear trend from the time series. +``` +Next, copy and paste the code below to estimate the linear trend using the linearRegression reducer, and remove that linear trend from the time series. +```js // List of the independent variable names. -var independents = ee.List(['constant', 't']); +var independents = ee.List(['constant', 't']); // Name of the dependent variable. -var dependent = ee.String('NDVI'); +var dependent = ee.String('NDVI'); -// Compute a linear trend.  This will have two bands: 'residuals' and +// Compute a linear trend. This will have two bands: 'residuals' and // a 2x1 band called coefficients (columns are for dependent variables). -var trend = landsat8sr.select(independents.add(dependent)) -   .reduce(ee.Reducer.linearRegression(independents.length(), 1)); +var trend = landsat8sr.select(independents.add(dependent)) + .reduce(ee.Reducer.linearRegression(independents.length(), 1)); // Flatten the coefficients into a 2-band image -var coefficients = trend.select('coefficients')   // Get rid of extra dimensions and convert back to a regular image   .arrayProject([0]) -   .arrayFlatten([independents]); +var coefficients = trend.select('coefficients') // Get rid of extra dimensions and convert back to a regular image .arrayProject([0]) + .arrayFlatten([independents]); // Compute a detrended series. -var detrended = landsat8sr.map(function(image) {   return image.select(dependent) -       .subtract(image.select(independents).multiply( -               coefficients) -           .reduce('sum')) -       .rename(dependent) -       .copyProperties(image, ['system:time_start']); +var detrended = landsat8sr.map(function(image) { return image.select(dependent) + .subtract(image.select(independents).multiply( + coefficients) + .reduce('sum')) + .rename(dependent) + .copyProperties(image, ['system:time_start']); }); -Now let’s turn to autocovariance and autocorrelation. The autocovariance of a time series refers to the dependence of values in the time series at time t with values at time h = t − lag. The autocorrelation is the correlation between elements of a dataset at one time and elements of the same dataset at a different time. The autocorrelation is the autocovariance normalized by the standard deviations of the covariates. Specifically, we assume our time series is stationary, and define the autocovariance and autocorrelation following Shumway and Stoffer (2019). Comparing values at time t to previous values is useful not only for computing autocovariance, but also for a variety of other time series analyses as you'll see shortly. +``` +Now let’s turn to autocovariance and autocorrelation. The autocovariance of a time series refers to the dependence of values in the time series at time t with values at time h = t − lag. The autocorrelation is the correlation between elements of a dataset at one time and elements of the same dataset at a different time. The autocorrelation is the autocovariance normalized by the standard deviations of the covariates. Specifically, we assume our time series is stationary, and define the autocovariance and autocorrelation following Shumway and Stoffer (2019). Comparing values at time t to previous values is useful not only for computing autocovariance, but also for a variety of other time series analyses as you'll see shortly. -To combine image data with previous values in Earth Engine, the first step is to join the previous values to the current values. To do that, we will use a ee.Join function to create what we'll call a lagged collection. Copy and paste the code below to define a function that creates a lagged collection. +To combine image data with previous values in Earth Engine, the first step is to join the previous values to the current values. To do that, we will use a ee.Join function to create what we'll call a lagged collection. Copy and paste the code below to define a function that creates a lagged collection. +```js // Function that creates a lagged collection. -var lag = function(leftCollection, rightCollection, lagDays) {   var filter = ee.Filter.and(       ee.Filter.maxDifference({ -           difference: 1000 * 60 * 60 * 24 * lagDays, -           leftField: 'system:time_start', -           rightField: 'system:time_start'       }),       ee.Filter.greaterThan({ -           leftField: 'system:time_start', -           rightField: 'system:time_start'       }));   return ee.Join.saveAll({ -       matchesKey: 'images', -       measureKey: 'delta_t', -       ordering: 'system:time_start', -       ascending: false, // Sort reverse chronologically   }).apply({ -       primary: leftCollection, -       secondary: rightCollection, -       condition: filter -   }); +var lag = function(leftCollection, rightCollection, lagDays) { var filter = ee.Filter.and( ee.Filter.maxDifference({ + difference: 1000 * 60 * 60 * 24 * lagDays, + leftField: 'system:time_start', + rightField: 'system:time_start' }), ee.Filter.greaterThan({ + leftField: 'system:time_start', + rightField: 'system:time_start' })); return ee.Join.saveAll({ + matchesKey: 'images', + measureKey: 'delta_t', + ordering: 'system:time_start', + ascending: false, // Sort reverse chronologically }).apply({ + primary: leftCollection, + secondary: rightCollection, + condition: filter + }); }; -This function joins a collection to itself, using a filter that gets all the images before each image’s date that are within a specified time difference (in days) of each image. That list of previous images within the lag time is stored in a property of the image called images, sorted reverse chronologically. For example, to create a lagged collection from the detrended Landsat imagery, copy and paste: +``` +This function joins a collection to itself, using a filter that gets all the images before each image’s date that are within a specified time difference (in days) of each image. That list of previous images within the lag time is stored in a property of the image called images, sorted reverse chronologically. For example, to create a lagged collection from the detrended Landsat imagery, copy and paste: +```js // Create a lagged collection of the detrended imagery. -var lagged17 = lag(detrended, detrended, 17); +var lagged17 = lag(detrended, detrended, 17); +``` Why 17 days? Recall that the temporal cadence of Landsat is 16 days. Specifying 17 days in the join gets one previous image, but no more. -Now, we will compute the autocovariance using a reducer that expects a set of one-dimensional arrays as input. So pixel values corresponding to time t need to be stacked with pixel values at time t − lag as multiple bands in the same image. Copy and paste the code below to define a function to do so, and apply it to merge the bands from the lagged collection. +Now, we will compute the autocovariance using a reducer that expects a set of one-dimensional arrays as input. So pixel values corresponding to time t need to be stacked with pixel values at time t − lag as multiple bands in the same image. Copy and paste the code below to define a function to do so, and apply it to merge the bands from the lagged collection. +```js // Function to stack bands. -var merge = function(image) {   // Function to be passed to iterate.   var merger = function(current, previous) {       return ee.Image(previous).addBands(current); -   };   return ee.ImageCollection.fromImages(image.get('images')) -       .iterate(merger, image); +var merge = function(image) { // Function to be passed to iterate. var merger = function(current, previous) { return ee.Image(previous).addBands(current); + }; return ee.ImageCollection.fromImages(image.get('images')) + .iterate(merger, image); }; // Apply merge function to the lagged collection. -var merged17 = ee.ImageCollection(lagged17.map(merge)); +var merged17 = ee.ImageCollection(lagged17.map(merge)); -Now the bands from time t and h are all in the same image. Note that the band name of a pixel at time h, ph, was the same as time t, pt (band name is “NDVI” in this case). During the merging process, it gets a '_1' appended to it (e.g. NDVI_1). +``` +Now the bands from time t and h are all in the same image. Note that the band name of a pixel at time h, ph, was the same as time t, pt (band name is “NDVI” in this case). During the merging process, it gets a '_1' appended to it (e.g. NDVI_1). -You can print the image collection to check the band names of one of the images. Copy and paste the code below to map a function to convert the merged bands to arrays with bands pt and ph, and then reduce it with the covariance reducer. We use a parallelScale factor of 8 in the reduce function to avoid the computation to run out of memory (this is not always needed). Note that the output of the covariance reducer is an array image, in which each pixel stores a 2x2 variance-covariance array. The off-diagonal elements are covariance, which you can map directly using the arrayGet function. +You can print the image collection to check the band names of one of the images. Copy and paste the code below to map a function to convert the merged bands to arrays with bands pt and ph, and then reduce it with the covariance reducer. We use a parallelScale factor of 8 in the reduce function to avoid the computation to run out of memory (this is not always needed). Note that the output of the covariance reducer is an array image, in which each pixel stores a 2x2 variance-covariance array. The off-diagonal elements are covariance, which you can map directly using the arrayGet function. +```js // Function to compute covariance. -var covariance = function(mergedCollection, band, lagBand) {   return mergedCollection.select([band, lagBand]).map(function( -       image) {       return image.toArray(); -   }).reduce(ee.Reducer.covariance(), 8); +var covariance = function(mergedCollection, band, lagBand) { return mergedCollection.select([band, lagBand]).map(function( + image) { return image.toArray(); + }).reduce(ee.Reducer.covariance(), 8); }; // Concatenate the suffix to the NDVI band. -var lagBand = dependent.cat('_1'); +var lagBand = dependent.cat('_1'); // Compute covariance. -var covariance17 = ee.Image(covariance(merged17, dependent, lagBand)) -   .clip(roi); +var covariance17 = ee.Image(covariance(merged17, dependent, lagBand)) + .clip(roi); // The output of the covariance reducer is an array image, // in which each pixel stores a 2x2 variance-covariance array. // The off diagonal elements are covariance, which you can map // directly using: Map.addLayer(covariance17.arrayGet([0, 1]), -   { -       min: 0, -       max: 0.02   },   'covariance (lag = 17 days)'); + { + min: 0, + max: 0.02 }, 'covariance (lag = 17 days)'); -Inspect the pixel values of the resulting covariance image (Fig. F4.9.1). The covariance is positive when the greater values of one variable (at time t) mainly correspond to the greater values of the other variable (at time h), and the same holds for the lesser values, therefore, the values tend to show similar behavior. In the opposite case, when the greater values of a variable correspond to the lesser values of the other variable, the covariance is negative. +``` +Inspect the pixel values of the resulting covariance image (Fig. F4.9.1). The covariance is positive when the greater values of one variable (at time t) mainly correspond to the greater values of the other variable (at time h), and the same holds for the lesser values, therefore, the values tend to show similar behavior. In the opposite case, when the greater values of a variable correspond to the lesser values of the other variable, the covariance is negative. -![](F4/image2.png) +![Fig. F4.9.1 Autocovariance image](F4/image2.png) -Fig. F4.9.1 Autocovariance image The diagonal elements of the variance-covariance array are variances. Copy and paste the code below to define and map a function to compute correlation (Fig. F4.9.2) from the variance-covariance array. +```js // Define the correlation function. -var correlation = function(vcArrayImage) {   var covariance = ee.Image(vcArrayImage).arrayGet([0, 1]);   var sd0 = ee.Image(vcArrayImage).arrayGet([0, 0]).sqrt();   var sd1 = ee.Image(vcArrayImage).arrayGet([1, 1]).sqrt();   return covariance.divide(sd0).divide(sd1).rename(       'correlation'); +var correlation = function(vcArrayImage) { var covariance = ee.Image(vcArrayImage).arrayGet([0, 1]); var sd0 = ee.Image(vcArrayImage).arrayGet([0, 0]).sqrt(); var sd1 = ee.Image(vcArrayImage).arrayGet([1, 1]).sqrt(); return covariance.divide(sd0).divide(sd1).rename( 'correlation'); }; // Apply the correlation function. -var correlation17 = correlation(covariance17).clip(roi); +var correlation17 = correlation(covariance17).clip(roi); Map.addLayer(correlation17, -   { -       min: -1, -       max: 1   },   'correlation (lag = 17 days)'); + { + min: -1, + max: 1 }, 'correlation (lag = 17 days)'); -![](F4/image33.png) +``` +![Fig. F4.9.2 Autocorrelation image](F4/image33.png) -Fig. F4.9.2 Autocorrelation image Higher positive values indicate higher correlation between the elements of the dataset, and lower negative values indicate the opposite. -It's worth noting that you can do this for longer lags as well. Of course, that images list will fill up with all the images that are within lag of t. Those other images are also useful—for example, in fitting autoregressive models as described later. +It's worth noting that you can do this for longer lags as well. Of course, that images list will fill up with all the images that are within lag of t. Those other images are also useful—for example, in fitting autoregressive models as described later. -::: {.callout-note} -Code Checkpoint F49a. The book’s repository contains a script that shows what your code should look like at this point. +:::{.callout-note} +Code Checkpoint F49a. The book’s repository contains a script that shows what your code should look like at this point. ::: ## Cross-Covariance and Cross-Correlation -Cross-covariance is analogous to autocovariance, except instead of measuring the correspondence between a variable and itself at a lag, it measures the correspondence between a variable and a covariate at a lag. Specifically, we will define the cross-covariance and cross-correlation according to Shumway and Stoffer (2019).  +Cross-covariance is analogous to autocovariance, except instead of measuring the correspondence between a variable and itself at a lag, it measures the correspondence between a variable and a covariate at a lag. Specifically, we will define the cross-covariance and cross-correlation according to Shumway and Stoffer (2019). You already have all the code needed to compute cross-covariance and cross-correlation. But you do need a time series of another variable. Suppose we postulate that NDVI is related in some way to the precipitation before the NDVI was observed. To estimate the strength of this relationship in every pixel, copy and paste the code below to the existing script to load precipitation, join, merge, and reduce as previously: +```js // Precipitation (covariate) -var chirps = ee.ImageCollection('UCSB-CHG/CHIRPS/PENTAD'); +var chirps = ee.ImageCollection('UCSB-CHG/CHIRPS/PENTAD'); // Join the t-l (l=1 pentad) precipitation images to the Landsat. -var lag1PrecipNDVI = lag(landsat8sr, chirps, 5); +var lag1PrecipNDVI = lag(landsat8sr, chirps, 5); // Add the precipitation images as bands. -var merged1PrecipNDVI = ee.ImageCollection(lag1PrecipNDVI.map(merge)); +var merged1PrecipNDVI = ee.ImageCollection(lag1PrecipNDVI.map(merge)); // Compute and display cross-covariance. -var cov1PrecipNDVI = covariance(merged1PrecipNDVI, 'NDVI',   'precipitation').clip(roi); -Map.addLayer(cov1PrecipNDVI.arrayGet([0, 1]), {},   'NDVI - PRECIP cov (lag = 5)'); +var cov1PrecipNDVI = covariance(merged1PrecipNDVI, 'NDVI', 'precipitation').clip(roi); +Map.addLayer(cov1PrecipNDVI.arrayGet([0, 1]), {}, 'NDVI - PRECIP cov (lag = 5)'); // Compute and display cross-correlation. -var corr1PrecipNDVI = correlation(cov1PrecipNDVI).clip(roi); +var corr1PrecipNDVI = correlation(cov1PrecipNDVI).clip(roi); Map.addLayer(corr1PrecipNDVI, { -   min: -0.5, -   max: 0.5}, 'NDVI - PRECIP corr (lag = 5)'); + min: -0.5, + max: 0.5}, 'NDVI - PRECIP corr (lag = 5)'); -What do you observe from this result? Looking at the cross-correlation image (Fig. F4.9.3), do you observe high values where you would expect high NDVI values (vegetated areas)? One possible drawback of this computation is that it's only based on five days of precipitation, whichever five days came right before the NDVI image. +``` +What do you observe from this result? Looking at the cross-correlation image (Fig. F4.9.3), do you observe high values where you would expect high NDVI values (vegetated areas)? One possible drawback of this computation is that it's only based on five days of precipitation, whichever five days came right before the NDVI image. -![](F4/image46.png) +![Fig. F4.9.3 Cross-correlation image of NDVI and precipitation with a five-day lag.](F4/image46.png) -Fig. F4.9.3 Cross-correlation image of NDVI and precipitation with a five-day lag. Perhaps precipitation in the month before the observed NDVI is relevant? Copy and paste the code below to test the 30-day lag idea. +```js // Join the precipitation images from the previous month. -var lag30PrecipNDVI = lag(landsat8sr, chirps, 30); +var lag30PrecipNDVI = lag(landsat8sr, chirps, 30); -var sum30PrecipNDVI = ee.ImageCollection(lag30PrecipNDVI.map(function( -   image) {   var laggedImages = ee.ImageCollection.fromImages(image -       .get('images'));   return ee.Image(image).addBands(laggedImages.sum() -       .rename('sum')); +var sum30PrecipNDVI = ee.ImageCollection(lag30PrecipNDVI.map(function( + image) { var laggedImages = ee.ImageCollection.fromImages(image + .get('images')); return ee.Image(image).addBands(laggedImages.sum() + .rename('sum')); })); // Compute covariance. -var cov30PrecipNDVI = covariance(sum30PrecipNDVI, 'NDVI', 'sum').clip( -   roi); -Map.addLayer(cov1PrecipNDVI.arrayGet([0, 1]), {},   'NDVI - sum cov (lag = 30)'); +var cov30PrecipNDVI = covariance(sum30PrecipNDVI, 'NDVI', 'sum').clip( + roi); +Map.addLayer(cov1PrecipNDVI.arrayGet([0, 1]), {}, 'NDVI - sum cov (lag = 30)'); // Correlation. -var corr30PrecipNDVI = correlation(cov30PrecipNDVI).clip(roi); +var corr30PrecipNDVI = correlation(cov30PrecipNDVI).clip(roi); Map.addLayer(corr30PrecipNDVI, { -   min: -0.5, -   max: 0.5}, 'NDVI - sum corr (lag = 30)'); + min: -0.5, + max: 0.5}, 'NDVI - sum corr (lag = 30)'); -Observe that the only change is to the merge method. Instead of merging the bands of the NDVI image and the covariate (precipitation), the entire list of precipitation is summed and added as a band (eliminating the need for iterate). +``` +Observe that the only change is to the merge method. Instead of merging the bands of the NDVI image and the covariate (precipitation), the entire list of precipitation is summed and added as a band (eliminating the need for iterate). -Which changes do you notice between the cross-correlation images—5 days lag vs. 30 days lag (Fig. F4.9.4)?. You can use the Inspector tool to assess if the correlation increased or not at vegetated areas. +Which changes do you notice between the cross-correlation images—5 days lag vs. 30 days lag (Fig. F4.9.4)?. You can use the Inspector tool to assess if the correlation increased or not at vegetated areas. -![](F4/image34.png) +![Fig. F4.9.4 Cross-correlation image of NDVI and precipitation with a 30-day lag.](F4/image34.png) -Fig. F4.9.4 Cross-correlation image of NDVI and precipitation with a 30-day lag. As long as there is sufficient temporal overlap between the time series, these techniques could be extended to longer lags and longer time series. -::: {.callout-note} -Code Checkpoint F49b. The book’s repository contains a script that shows what your code should look like at this point. +:::{.callout-note} +Code Checkpoint F49b. The book’s repository contains a script that shows what your code should look like at this point. ::: ## Auto-Regressive Models -The discussion of autocovariance preceded this section in order to introduce the concept of lag. Now that you have a way to get previous values of a variable, it's worth considering auto-regressive models. Suppose that pixel values at time t depend in some way on previous pixel values—auto-regressive models are time series models that use observations from previous time steps as input to a regression equation to predict the value at the next time step. If you have observed significant, non-zero autocorrelations in a time series, this is a good assumption. Specifically, you may postulate a linear model such as the following, where pt is a pixel at time t, and et is a random error (Chap. F4.6): +The discussion of autocovariance preceded this section in order to introduce the concept of lag. Now that you have a way to get previous values of a variable, it's worth considering auto-regressive models. Suppose that pixel values at time t depend in some way on previous pixel values—auto-regressive models are time series models that use observations from previous time steps as input to a regression equation to predict the value at the next time step. If you have observed significant, non-zero autocorrelations in a time series, this is a good assumption. Specifically, you may postulate a linear model such as the following, where pt is a pixel at time t, and et is a random error (Chap. F4.6): -pt = β0 + β1pt-1 + β2pt-2 + et                                        (F4.9.1) +pt = β0 + β1pt-1 + β2pt-2 + et (F4.9.1) -To fit this model, you need a lagged collection as created previously, except with a longer lag (e.g., lag = 34 days). The next steps are to merge the bands, then reduce with the linear regression reducer. +To fit this model, you need a lagged collection as created previously, except with a longer lag (e.g., lag = 34 days). The next steps are to merge the bands, then reduce with the linear regression reducer. -Copy and paste the line below to the existing script to create a lagged collection, where the images list stores the two previous images: +Copy and paste the line below to the existing script to create a lagged collection, where the images list stores the two previous images: -var lagged34 = ee.ImageCollection(lag(landsat8sr, landsat8sr, 34)); +var lagged34 = ee.ImageCollection(lag(landsat8sr, landsat8sr, 34)); -Copy and paste the code below to merge the bands of the lagged collection such that each image has bands at time t and bands at times t - 1,..., t − lag. Note that it's necessary to filter out any images that don't have two previous temporal neighbors. +Copy and paste the code below to merge the bands of the lagged collection such that each image has bands at time t and bands at times t - 1,..., t − lag. Note that it's necessary to filter out any images that don't have two previous temporal neighbors. -var merged34 = lagged34.map(merge).map(function(image) {   return image.set('n', ee.List(image.get('images')) -       .length()); +var merged34 = lagged34.map(merge).map(function(image) { return image.set('n', ee.List(image.get('images')) + .length()); }).filter(ee.Filter.gt('n', 1)); -Now, copy and paste the code below to fit the regression model using the linearRegression reducer. +Now, copy and paste the code below to fit the regression model using the linearRegression reducer. -var arIndependents = ee.List(['constant', 'NDVI_1', 'NDVI_2']); +var arIndependents = ee.List(['constant', 'NDVI_1', 'NDVI_2']); -var ar2 = merged34 -   .select(arIndependents.add(dependent)) -   .reduce(ee.Reducer.linearRegression(arIndependents.length(), 1)); +var ar2 = merged34 + .select(arIndependents.add(dependent)) + .reduce(ee.Reducer.linearRegression(arIndependents.length(), 1)); +```js // Turn the array image into a multi-band image of coefficients. -var arCoefficients = ar2.select('coefficients') -   .arrayProject([0]) -   .arrayFlatten([arIndependents]); +var arCoefficients = ar2.select('coefficients') + .arrayProject([0]) + .arrayFlatten([arIndependents]); -We can compute the fitted values using the expression function in Earth Engine. Because this model is a function of previous pixel values, which may be masked, if any of the inputs to equation F4.9.1 are masked, the output of the equation will also be masked. That's why you should use an expression here, unlike the previous linear models of time. Copy and paste the code below to compute the fitted values. +``` +We can compute the fitted values using the expression function in Earth Engine. Because this model is a function of previous pixel values, which may be masked, if any of the inputs to equation F4.9.1 are masked, the output of the equation will also be masked. That's why you should use an expression here, unlike the previous linear models of time. Copy and paste the code below to compute the fitted values. +```js // Compute fitted values. -var fittedAR = merged34.map(function(image) {   return image.addBands( -       image.expression(           'beta0 + beta1 * p1 + beta2 * p2', { -               p1: image.select('NDVI_1'), -               p2: image.select('NDVI_2'), -               beta0: arCoefficients.select('constant'), -               beta1: arCoefficients.select('NDVI_1'), -               beta2: arCoefficients.select('NDVI_2') -           }).rename('fitted')); +var fittedAR = merged34.map(function(image) { return image.addBands( + image.expression( 'beta0 + beta1 * p1 + beta2 * p2', { + p1: image.select('NDVI_1'), + p2: image.select('NDVI_2'), + beta0: arCoefficients.select('constant'), + beta1: arCoefficients.select('NDVI_1'), + beta2: arCoefficients.select('NDVI_2') + }).rename('fitted')); }); -Finally, copy and paste the code below to plot the results (Fig. F4.9.5). We will use a specific point defined as pt. Note the missing values that result from masked data. If you run into computation errors, try commenting the Map.addLayer calls from previous sections to save memory. +``` +Finally, copy and paste the code below to plot the results (Fig. F4.9.5). We will use a specific point defined as pt. Note the missing values that result from masked data. If you run into computation errors, try commenting the Map.addLayer calls from previous sections to save memory. +```js // Create an Earth Engine point object to print the time series chart. -var pt = ee.Geometry.Point([-119.0955, 35.9909]); +var pt = ee.Geometry.Point([-119.0955, 35.9909]); print(ui.Chart.image.series( -       fittedAR.select(['fitted', 'NDVI']), pt, ee.Reducer -   .mean(), 30) -   .setSeriesNames(['NDVI', 'fitted']) -   .setOptions({ -       title: 'AR(2) model: original and fitted values', -       lineWidth: 1, -       pointSize: 3, -   })); + fittedAR.select(['fitted', 'NDVI']), pt, ee.Reducer + .mean(), 30) + .setSeriesNames(['NDVI', 'fitted']) + .setOptions({ + title: 'AR(2) model: original and fitted values', + lineWidth: 1, + pointSize: 3, + })); -![](F4/image19.png) +``` +![Fig. F4.9.5 Observed NDVI and fitted values at selected point](F4/image19.png) -Fig. F4.9.5 Observed NDVI and fitted values at selected point At this stage, note that the missing data has become a real problem. Any data point for which at least one of the previous points is masked or missing is also masked. -::: {.callout-note} -Code Checkpoint F49c. The book’s repository contains a script that shows what your code should look like at this point. +:::{.callout-note} +Code Checkpoint F49c. The book’s repository contains a script that shows what your code should look like at this point. ::: It may be possible to avoid this problem by substituting the output from equation F4.9.1 (the modeled value) for the missing or masked data. Unfortunately, the code to make that happen is not straightforward. You can check a solution in the following Code Checkpoint: -::: {.callout-note} -Code Checkpoint F49d. The book’s repository contains a script that shows what your code should look like at this point. +:::{.callout-note} +Code Checkpoint F49d. The book’s repository contains a script that shows what your code should look like at this point. ::: ## Synthesis {.unnumbered} -Assignment 1. Analyze cross-correlation between NDVI and soil moisture, or precipitation and soil moisture, for example. Earth Engine contains different soil moisture datasets in its catalog (e.g., NASA-USDA SMAP, NASA-GLDAS). Try increasing the lagged time and see if it makes any difference. Alternatively, you can pick any other environmental variable/index (e.g., a different vegetation index: EVI instead of NDVI, for example) and analyze its autocorrelation. +Assignment 1. Analyze cross-correlation between NDVI and soil moisture, or precipitation and soil moisture, for example. Earth Engine contains different soil moisture datasets in its catalog (e.g., NASA-USDA SMAP, NASA-GLDAS). Try increasing the lagged time and see if it makes any difference. Alternatively, you can pick any other environmental variable/index (e.g., a different vegetation index: EVI instead of NDVI, for example) and analyze its autocorrelation. ## Conclusion {.unnumbered} diff --git a/F5.qmd b/F5.qmd index 826b0b3..3124647 100644 --- a/F5.qmd +++ b/F5.qmd @@ -13,7 +13,7 @@ In addition to raster data processing, Earth Engine supports a rich set of vecto -::: {.callout-tip} +:::{.callout-tip} # Chapter Information ## Author {.unlisted .unnumbered} @@ -25,9 +25,9 @@ AJ Purdy, Ellen Brock, David Saah Overview -In this chapter, you will learn about features and feature collections and how to use them in conjunction with images and image collections in Earth Engine. Maps are useful for understanding spatial patterns, but scientists often need to extract statistics to answer a question. For example, you may make a false-color composite showing which areas of San Francisco are more “green”—i.e., have more healthy vegetation—than others, but you will likely not be able to directly determine which block in a neighborhood is the most green. This tutorial will demonstrate how to do just that by utilizing vectors. +In this chapter, you will learn about features and feature collections and how to use them in conjunction with images and image collections in Earth Engine. Maps are useful for understanding spatial patterns, but scientists often need to extract statistics to answer a question. For example, you may make a false-color composite showing which areas of San Francisco are more “green”—i.e., have more healthy vegetation—than others, but you will likely not be able to directly determine which block in a neighborhood is the most green. This tutorial will demonstrate how to do just that by utilizing vectors. -As described in Chap. F4.0, an important way to summarize and simplify data in Earth Engine is through the use of reducers. Reducers operating across space were used in Chap. F3.0, for example, to enable image regression between bands. More generally, chapters in Part F3 and Part F4 used reducers mostly to summarize the values across bands or images on a pixel-by-pixel basis. What if you wanted to summarize information within the confines of given spatial elements- for example, within a set of polygons? In this chapter, we will illustrate and explore Earth Engine’s method for doing that, which is through a reduceRegions call. +As described in Chap. F4.0, an important way to summarize and simplify data in Earth Engine is through the use of reducers. Reducers operating across space were used in Chap. F3.0, for example, to enable image regression between bands. More generally, chapters in Part F3 and Part F4 used reducers mostly to summarize the values across bands or images on a pixel-by-pixel basis. What if you wanted to summarize information within the confines of given spatial elements- for example, within a set of polygons? In this chapter, we will illustrate and explore Earth Engine’s method for doing that, which is through a reduceRegions call. Learning Outcomes @@ -35,7 +35,7 @@ Learning Outcomes * Creating a new feature using the geometry tools. * Importing and filtering a feature collection in Earth Engine. * Using a feature to clip and reduce image values within a geometry. -* Use reduceRegions to summarize an image in irregular neighborhoods. +* Use reduceRegions to summarize an image in irregular neighborhoods. * Exporting calculated data to tables with Tasks. Assumes you know how to: @@ -44,53 +44,53 @@ Assumes you know how to: * Calculate and interpret vegetation indices (Chap. F2.0). * Use drawing tools to create points, lines, and polygons (Chap. F2.1). -Introduction to Theory  +Introduction to Theory -In the world of geographic information systems (GIS), data is typically thought of in one of two basic data structures: raster and vector. In previous chapters, we have principally been focused on raster data—data using the remote sensing vocabulary of pixels, spatial resolution, images, and image collections. Working within the vector framework is also a crucial skill to master. If you don’t know much about GIS, you can find any number of online explainers of the distinctions between these data types, their strengths and limitations, and analyses using both data types. Being able to move fluidly between a raster conception and a vector conception of the world is powerful, and is facilitated with specialized functions and approaches in Earth Engine.     +In the world of geographic information systems (GIS), data is typically thought of in one of two basic data structures: raster and vector. In previous chapters, we have principally been focused on raster data—data using the remote sensing vocabulary of pixels, spatial resolution, images, and image collections. Working within the vector framework is also a crucial skill to master. If you don’t know much about GIS, you can find any number of online explainers of the distinctions between these data types, their strengths and limitations, and analyses using both data types. Being able to move fluidly between a raster conception and a vector conception of the world is powerful, and is facilitated with specialized functions and approaches in Earth Engine. For our purposes, you can think of vector data as information represented as points (e.g., locations of sample sites), lines (e.g., railroad tracks), or polygons (e.g., the boundary of a national park or a neighborhood). Line data and polygon data are built up from points: for example, the latitude and longitude of the sample sites, the points along the curve of the railroad tracks, and the corners of the park that form its boundary. These points each have a highly specific location on Earth’s surface, and the vector data formed from them can be used for calculations with respect to other layers. As will be seen in this chapter, for example, a polygon can be used to identify which pixels in an image are contained within its borders. Point-based data have already been used in earlier chapters for filtering image collections by location (see Part F1), and can also be used to extract values from an image at a point or a set of points (see Chap. F5.2). Lines possess the dimension of length and have similar capabilities for filtering image collections and accessing their values along a transect. In addition to using polygons to summarize values within a boundary, they can be used for other, similar purposes—for example, to clip an image. -As you have seen, raster features in Earth Engine are stored as an Image or as part of an ImageCollection. Using a similar conceptual model, vector data in Earth Engine is stored as a Feature or as part of a FeatureCollection. Features and feature collections provide useful data to filter images and image collections by their location, clip images to a boundary, or statistically summarize the pixel values within a region. +As you have seen, raster features in Earth Engine are stored as an Image or as part of an ImageCollection. Using a similar conceptual model, vector data in Earth Engine is stored as a Feature or as part of a FeatureCollection. Features and feature collections provide useful data to filter images and image collections by their location, clip images to a boundary, or statistically summarize the pixel values within a region. -In the following example, you will use features and feature collections to identify which city block near the University of San Francisco (USF) campus is the most green. +In the following example, you will use features and feature collections to identify which city block near the University of San Francisco (USF) campus is the most green. -## Using Geometry Tools to Create Features in Earth Engine +## Using Geometry Tools to Create Features in Earth Engine To demonstrate how geometry tools in Earth Engine work, let’s start by creating a point, and two polygons to represent different elements on the USF campus. -Click on the geometry tools in the top left of the Map pane and create a point feature. Place a new point where USF is located (see Fig. F5.0.1). +Click on the geometry tools in the top left of the Map pane and create a point feature. Place a new point where USF is located (see Fig. F5.0.1). -![](F5/image54.png) +![Fig. F5.0.1 Location of the USF campus in San Francisco, California. Your first point should be in this vicinity. The red arrow points to the geometry tools.](F5/image54.png) -Fig. F5.0.1 Location of the USF campus in San Francisco, California. Your first point should be in this vicinity. The red arrow points to the geometry tools. -Use Google Maps to search for “Harney Science Center” or “Lo Schiavo Center for Science.” Hover your mouse over the Geometry Imports to find the +new layer menu item and add a new layer to delineate the boundary of a building on campus. +Use Google Maps to search for “Harney Science Center” or “Lo Schiavo Center for Science.” Hover your mouse over the Geometry Imports to find the +new layer menu item and add a new layer to delineate the boundary of a building on campus. Next, create another new layer to represent the entire campus as a polygon. -After you create these layers, rename the geometry imports at the top of your script. Name the layers usf_point, usf_building, and usf_campus. These names are used within the script shown in Fig. F5.0.2. +After you create these layers, rename the geometry imports at the top of your script. Name the layers usf_point, usf_building, and usf_campus. These names are used within the script shown in Fig. F5.0.2. -![](F5/image10.png) +![Fig. F5.0.2 Rename the default variable names for each layer in the Imports section of the code at the top of your script](F5/image10.png) -Fig. F5.0.2 Rename the default variable names for each layer in the Imports section of the code at the top of your script -::: {.callout-note} -Code Checkpoint F50a. The book’s repository contains a script that shows what your code should look like at this point.  +:::{.callout-note} +Code Checkpoint F50a. The book’s repository contains a script that shows what your code should look like at this point. ::: -## Loading Existing Features and Feature Collections in Earth Engine +## Loading Existing Features and Feature Collections in Earth Engine If you wish to have the exact same geometry imports in this chapter for the rest of this exercise, begin this section using the code at the Code Checkpoint above. -Next, you will load a city block dataset to determine the amount of vegetation on blocks near USF. The code below imports an existing feature dataset in Earth Engine. The Topologically Integrated Geographic Encoding and Referencing (TIGER) boundaries are census-designated boundaries that are a useful resource when comparing socioeconomic and diversity metrics with environmental datasets in the United States. +Next, you will load a city block dataset to determine the amount of vegetation on blocks near USF. The code below imports an existing feature dataset in Earth Engine. The Topologically Integrated Geographic Encoding and Referencing (TIGER) boundaries are census-designated boundaries that are a useful resource when comparing socioeconomic and diversity metrics with environmental datasets in the United States. +```js // Import the Census Tiger Boundaries from GEE. -var tiger = ee.FeatureCollection('TIGER/2010/Blocks'); +var tiger = ee.FeatureCollection('TIGER/2010/Blocks'); // Add the new feature collection to the map, but do not display. -Map.addLayer(tiger, {   'color': 'black'}, 'Tiger', false); +Map.addLayer(tiger, { 'color': 'black'}, 'Tiger', false); -You should now have the geometry for USF’s campus and a layer added to your map that is not visualized for census blocks across the United States. Next, we will use neighborhood data to spatially filter the TIGER feature collection for blocks near USF’s campus. +``` +You should now have the geometry for USF’s campus and a layer added to your map that is not visualized for census blocks across the United States. Next, we will use neighborhood data to spatially filter the TIGER feature collection for blocks near USF’s campus. ## Importing Features into Earth Engine @@ -100,135 +100,142 @@ There are many image collections loaded in Earth Engine, and they can cover a ve Use your internet searching skills to locate the “Analysis Neighborhoods” dataset covering San Francisco. This data might be located in a number of places, including DataSF, the City of San Francisco’s public-facing data repository. -![](F5/image27.png) +![Fig. F5.0.3 DataSF website neighborhood shapefile to download](F5/image27.png) -Fig. F5.0.3 DataSF website neighborhood shapefile to download -After you find the Analysis Neighborhoods layer, click Export and select Shapefile (Fig. F5.0.3). Keep track of where you save the zipped file, as we will load this into Earth Engine. Shapefiles contain vector-based data—points, lines, polygons—and include a number of files, such as the location information, attribute information, and others. +After you find the Analysis Neighborhoods layer, click Export and select Shapefile (Fig. F5.0.3). Keep track of where you save the zipped file, as we will load this into Earth Engine. Shapefiles contain vector-based data—points, lines, polygons—and include a number of files, such as the location information, attribute information, and others. -Extract the folder to your computer. When you open the folder, you will see that there are actually many files. The extensions (.shp, .dbf, .shx, .prj) all provide a different piece of information to display vector-based data. The .shp file provides data on the geometry. The .dbf file provides data about the attributes. The .shx file is an index file. Lastly, the .prj file describes the map projection of the coordinate information for the shapefile. You will need to load all four files to create a new feature asset in Earth Engine. +Extract the folder to your computer. When you open the folder, you will see that there are actually many files. The extensions (.shp, .dbf, .shx, .prj) all provide a different piece of information to display vector-based data. The .shp file provides data on the geometry. The .dbf file provides data about the attributes. The .shx file is an index file. Lastly, the .prj file describes the map projection of the coordinate information for the shapefile. You will need to load all four files to create a new feature asset in Earth Engine. ### Upload SF Neighborhoods File as an Asset -Navigate to the Assets tab (near Scripts). Select New > Table Upload > Shape files (Fig. F5.0.4). +Navigate to the Assets tab (near Scripts). Select New > Table Upload > Shape files (Fig. F5.0.4). -![](F5/image52.png) +![Fig. F5.0.4 Import an asset as a zipped folder](F5/image52.png) -Fig. F5.0.4 Import an asset as a zipped folder ### Select Files and Name Asset -Click the Select button and then use the file navigator to select the component files of the shapefile structure (i.e., .shp, .dbf, .shx, and .prj) (Fig. F5.0.5). Assign an Asset Name so you can recognize this asset. +Click the Select button and then use the file navigator to select the component files of the shapefile structure (i.e., .shp, .dbf, .shx, and .prj) (Fig. F5.0.5). Assign an Asset Name so you can recognize this asset. -![](F5/image43.png) +![Fig. F5.0.5 Select the four files extracted from the zipped folder. Make sure each file has the same name and that there are no spaces in the file names of the component files of the shapefile structure.](F5/image43.png) -Fig. F5.0.5 Select the four files extracted from the zipped folder. Make sure each file has the same name and that there are no spaces in the file names of the component files of the shapefile structure. -Uploading the asset may take a few minutes. The status of the upload is presented under the Tasks tab. After your asset has been successfully loaded, click on the asset in the Assets folder and find the collection ID. Copy this text and use it to import the file into your Earth Engine analysis. +Uploading the asset may take a few minutes. The status of the upload is presented under the Tasks tab. After your asset has been successfully loaded, click on the asset in the Assets folder and find the collection ID. Copy this text and use it to import the file into your Earth Engine analysis. -Assign the asset to the table (collection) ID using the script below. Note that you will need to replace 'path/to/your/asset/assetname' with the actual path copied in the previous step. +Assign the asset to the table (collection) ID using the script below. Note that you will need to replace 'path/to/your/asset/assetname' with the actual path copied in the previous step. +```js // Assign the feature collection to the variable sfNeighborhoods. -var sfNeighborhoods = ee.FeatureCollection(   'path/to/your/asset/assetname'); +var sfNeighborhoods = ee.FeatureCollection( 'path/to/your/asset/assetname'); // Print the size of the feature collection. // (Answers the question how many features?) print(sfNeighborhoods.size()); -Map.addLayer(sfNeighborhoods, {   'color': 'blue'}, 'sfNeighborhoods'); +Map.addLayer(sfNeighborhoods, { 'color': 'blue'}, 'sfNeighborhoods'); -Note that if you have any trouble with loading the FeatureCollection using the technique above, you can follow directions in the Checkpoint script below to use an existing asset loaded for this exercise. +``` +Note that if you have any trouble with loading the FeatureCollection using the technique above, you can follow directions in the Checkpoint script below to use an existing asset loaded for this exercise. -::: {.callout-note} -Code Checkpoint F50b. The book’s repository contains a script that shows what your code should look like at this point. +:::{.callout-note} +Code Checkpoint F50b. The book’s repository contains a script that shows what your code should look like at this point. ::: ## Filtering Feature Collections by Attributes ### Filter by Geometry of Another Feature -First, let’s find the neighborhood associated with USF. Use the first point you created to find the neighborhood that intersects this point; filterBounds is the tool that does that, returning a filtered feature. +First, let’s find the neighborhood associated with USF. Use the first point you created to find the neighborhood that intersects this point; filterBounds is the tool that does that, returning a filtered feature. +```js // Filter sfNeighborhoods by USF. -var usfNeighborhood = sfNeighborhoods.filterBounds(usf_point); +var usfNeighborhood = sfNeighborhoods.filterBounds(usf_point); +``` Now, filter the blocks layer by USF’s neighborhood and visualize it on the map. +```js // Filter the Census blocks by the boundary of the neighborhood layer. -var usfTiger = tiger.filterBounds(usfNeighborhood); +var usfTiger = tiger.filterBounds(usfNeighborhood); Map.addLayer(usfTiger, {}, 'usf_Tiger'); +``` ### Filter by Feature (Attribute) Properties -In addition to filtering a FeatureCollection by the location of another feature, you can also filter it by its properties. First, let’s print the usfTiger variable to the Console and inspect the object. +In addition to filtering a FeatureCollection by the location of another feature, you can also filter it by its properties. First, let’s print the usfTiger variable to the Console and inspect the object. print(usfTiger); -You can click on the feature collection name in the Console to uncover more information about the dataset. Click on the columns to learn about what attribute information is contained in this dataset. You will notice this feature collection contains information on both housing ('housing10') and population ('pop10'). +You can click on the feature collection name in the Console to uncover more information about the dataset. Click on the columns to learn about what attribute information is contained in this dataset. You will notice this feature collection contains information on both housing ('housing10') and population ('pop10'). Now you will filter for blocks with just the right amount of housing units. You don’t want it too dense, nor do you want too few neighbors. Filter the blocks to have fewer than 250 housing units. +```js // Filter for census blocks by housing units. -var housing10_l250 = usfTiger -   .filter(ee.Filter.lt('housing10', 250)); +var housing10_l250 = usfTiger + .filter(ee.Filter.lt('housing10', 250)); +``` Now filter the already-filtered blocks to have more than 50 housing units. -var housing10_g50_l250 = housing10_l250.filter(ee.Filter.gt(   'housing10', 50)); +var housing10_g50_l250 = housing10_l250.filter(ee.Filter.gt( 'housing10', 50)); Now, let’s visualize what this looks like. -Map.addLayer(housing10_g50_l250, {   'color': 'Magenta'}, 'housing'); +Map.addLayer(housing10_g50_l250, { 'color': 'Magenta'}, 'housing'); -We have combined spatial and attribute information to narrow the set to only those blocks that meet our criteria of having between 50 and 250 housing units. +We have combined spatial and attribute information to narrow the set to only those blocks that meet our criteria of having between 50 and 250 housing units. ### Print Feature (Attribute) Properties to Console -We can print out attribute information about these features. The block of code below prints out the area of the resultant geometry in square meters. +We can print out attribute information about these features. The block of code below prints out the area of the resultant geometry in square meters. -var housing_area = housing10_g50_l250.geometry().area(); +var housing_area = housing10_g50_l250.geometry().area(); print('housing_area:', housing_area); -The next block of code reduces attribute information and prints out the mean of the housing10 column. +The next block of code reduces attribute information and prints out the mean of the housing10 column. -var housing10_mean = usfTiger.reduceColumns({ -   reducer: ee.Reducer.mean(), -   selectors: ['housing10'] +var housing10_mean = usfTiger.reduceColumns({ + reducer: ee.Reducer.mean(), + selectors: ['housing10'] }); print('housing10_mean', housing10_mean); Both of the above sections of code provide meaningful information about each feature, but they do not tell us which block is the most green. The next section will address that question. -::: {.callout-note} -Code Checkpoint F50c. The book’s repository contains a script that shows what your code should look like at this point.  +:::{.callout-note} +Code Checkpoint F50c. The book’s repository contains a script that shows what your code should look like at this point. ::: ## Reducing Images Using Feature Geometry Now that we have identified the blocks around USF’s campus that have the right housing density, let’s find which blocks are the greenest. -The Normalized Difference Vegetation Index (NDVI), presented in detail in Chap. F2.0, is often used to compare the greenness of pixels in different locations. Values on land range from 0 to 1, with values closer to 1 representing healthier and greener vegetation than values near 0. +The Normalized Difference Vegetation Index (NDVI), presented in detail in Chap. F2.0, is often used to compare the greenness of pixels in different locations. Values on land range from 0 to 1, with values closer to 1 representing healthier and greener vegetation than values near 0. ### Create an NDVI Image -The code below imports the Landsat 8 ImageCollection as landsat8. Then, the code filters for images in 2021. Lastly, the code sorts the images from 2021 to find the least cloudy day. +The code below imports the Landsat 8 ImageCollection as landsat8. Then, the code filters for images in 2021. Lastly, the code sorts the images from 2021 to find the least cloudy day. +```js // Import the Landsat 8 TOA image collection. -var landsat8 = ee.ImageCollection('LANDSAT/LC08/C02/T1_TOA'); +var landsat8 = ee.ImageCollection('LANDSAT/LC08/C02/T1_TOA'); // Get the least cloudy image in 2015. -var image = ee.Image( -   landsat8 -   .filterBounds(usf_point) -   .filterDate('2015-01-01', '2015-12-31') -   .sort('CLOUD_COVER') -   .first()); +var image = ee.Image( + landsat8 + .filterBounds(usf_point) + .filterDate('2015-01-01', '2015-12-31') + .sort('CLOUD_COVER') + .first()); -The next section of code assigns the near-infrared band (B5) to variable nir and assigns the red band (B4) to red. Then the bands are combined together to compute NDVI as (nir − red)/(nir + red). +``` +The next section of code assigns the near-infrared band (B5) to variable nir and assigns the red band (B4) to red. Then the bands are combined together to compute NDVI as (nir − red)/(nir + red). -var nir = image.select('B5'); -var red = image.select('B4'); -var ndvi = nir.subtract(red).divide(nir.add(red)).rename('NDVI'); +var nir = image.select('B5'); +var red = image.select('B4'); +var ndvi = nir.subtract(red).divide(nir.add(red)).rename('NDVI'); ### Clip the NDVI Image to the Blocks Near USF @@ -236,53 +243,56 @@ Next, you will clip the NDVI layer to only show NDVI over USF’s neighborhood. The first section of code provides visualization settings. -var ndviParams = { -   min: -1, -   max: 1, -   palette: ['blue', 'white', 'green'] +var ndviParams = { + min: -1, + max: 1, + palette: ['blue', 'white', 'green'] }; The second block of code clips the image to our filtered housing layer. -var ndviUSFblocks = ndvi.clip(housing10_g50_l250); +var ndviUSFblocks = ndvi.clip(housing10_g50_l250); Map.addLayer(ndviUSFblocks, ndviParams, 'NDVI image'); Map.centerObject(usf_point, 14); -The NDVI map for all of San Francisco is interesting, and shows variability across the region. Now, let’s compute mean NDVI values for each block of the city. +The NDVI map for all of San Francisco is interesting, and shows variability across the region. Now, let’s compute mean NDVI values for each block of the city. ### Compute NDVI Statistics by Block -The code below uses the clipped image ndviUSFblocks and computes the mean NDVI value within each boundary. The scale provides a spatial resolution for the mean values to be computed on. +The code below uses the clipped image ndviUSFblocks and computes the mean NDVI value within each boundary. The scale provides a spatial resolution for the mean values to be computed on. +```js // Reduce image by feature to compute a statistic e.g. mean, max, min etc. -var ndviPerBlock = ndviUSFblocks.reduceRegions({ -   collection: housing10_g50_l250, -   reducer: ee.Reducer.mean(), -   scale: 30, +var ndviPerBlock = ndviUSFblocks.reduceRegions({ + collection: housing10_g50_l250, + reducer: ee.Reducer.mean(), + scale: 30, }); -Now we’ll use Earth Engine to find out which block is greenest.   +``` +Now we’ll use Earth Engine to find out which block is greenest. ### Export Table of NDVI Data by Block from Earth Engine to Google Drive Just as we loaded a feature into Earth Engine, we can export information from Earth Engine. Here, we will export the NDVI data, summarized by block, from Earth Engine to a Google Drive space so that we can interpret it in a program like Google Sheets or Excel. +```js // Get a table of data out of Google Earth Engine. Export.table.toDrive({ -   collection: ndviPerBlock, -   description: 'NDVI_by_block_near_USF' + collection: ndviPerBlock, + description: 'NDVI_by_block_near_USF' }); -When you run this code, you will notice that you have the Tasks tab highlighted on the top right of the Earth Engine Code Editor (Fig. F5.0.6). You will be prompted to name the directory when exporting the data. +``` +When you run this code, you will notice that you have the Tasks tab highlighted on the top right of the Earth Engine Code Editor (Fig. F5.0.6). You will be prompted to name the directory when exporting the data. -![](F5/image4.png) +![Fig. F5.0.6 Under the Tasks tab, select Run to initiate download](F5/image4.png) -Fig. F5.0.6 Under the Tasks tab, select Run to initiate download After you run the task, the file will be saved to your Google Drive. You have now brought a feature into Earth Engine and also exported data from Earth Engine. -::: {.callout-note} -Code Checkpoint F50d. The book’s repository contains a script that shows what your code should look like at this point.  +:::{.callout-note} +Code Checkpoint F50d. The book’s repository contains a script that shows what your code should look like at this point. ::: ## Identifying the Block in the Neighborhood Surrounding USF with the Highest NDVI @@ -290,33 +300,35 @@ You are already familiar with filtering datasets by their attributes. Now you wi ndviPerBlock = ndviPerBlock.select(['blockid10', 'mean']); print('ndviPerBlock', ndviPerBlock); -var ndviPerBlockSorted = ndviPerBlock.sort('mean', false); -var ndviPerBlockSortedFirst = ee.Feature(ndviPerBlock.sort('mean',       false) //Sort by NDVI mean in descending order.   .first()); //Select the block with the highest NDVI. +var ndviPerBlockSorted = ndviPerBlock.sort('mean', false); +var ndviPerBlockSortedFirst = ee.Feature(ndviPerBlock.sort('mean', false) //Sort by NDVI mean in descending order. .first()); //Select the block with the highest NDVI. print('ndviPerBlockSortedFirst', ndviPerBlockSortedFirst); -If you expand the feature of ndviPerBlockSortedFirst in the Console, you will be able to identify the blockid10 value of the greenest block and the mean NDVI value for that area. +If you expand the feature of ndviPerBlockSortedFirst in the Console, you will be able to identify the blockid10 value of the greenest block and the mean NDVI value for that area. -Another way to look at the data is by exporting the data to a table. Open the table using Google Sheets or Excel. You should see a column titled “mean.” Sort the mean column in descending order from highest NDVI to lowest NDVI, then use the blockid10 attribute to filter our feature collection one last time and display the greenest block near USF. +Another way to look at the data is by exporting the data to a table. Open the table using Google Sheets or Excel. You should see a column titled “mean.” Sort the mean column in descending order from highest NDVI to lowest NDVI, then use the blockid10 attribute to filter our feature collection one last time and display the greenest block near USF. +```js // Now filter by block and show on map! -var GreenHousing = usfTiger.filter(ee.Filter.eq('blockid10', +var GreenHousing = usfTiger.filter(ee.Filter.eq('blockid10', '###')); //< Put your id here prepend a 0! -Map.addLayer(GreenHousing, {   'color': 'yellow'}, 'Green Housing!'); +Map.addLayer(GreenHousing, { 'color': 'yellow'}, 'Green Housing!'); -::: {.callout-note} -Code Checkpoint F50e. The book’s repository contains a script that shows what your code should look like at this point.  +``` +:::{.callout-note} +Code Checkpoint F50e. The book’s repository contains a script that shows what your code should look like at this point. ::: ## Synthesis {.unnumbered} Now it’s your turn to use both feature classes and to reduce data using a geographic boundary. Create a new script for an area of interest and accomplish the following assignments. -Assignment 1. Create a study area map zoomed to a certain feature class that you made. +Assignment 1. Create a study area map zoomed to a certain feature class that you made. -Assignment 2. Filter one feature collection using feature properties. +Assignment 2. Filter one feature collection using feature properties. -Assignment 3. Filter one feature collection based on another feature’s location in space. +Assignment 3. Filter one feature collection based on another feature’s location in space. -Assignment 4. Reduce one image to the geometry of a feature in some capacity; e.g., extract a mean value or a value at a point. +Assignment 4. Reduce one image to the geometry of a feature in some capacity; e.g., extract a mean value or a value at a point. ## Conclusion {.unnumbered} @@ -330,7 +342,7 @@ In this chapter, you learned how to import features into Earth Engine. In Sect. -::: {.callout-tip} +:::{.callout-tip} # Chapter Information ## Author {.unlisted .unnumbered} @@ -342,7 +354,7 @@ Keiko Nomura, Samuel Bowers ## Overview {.unlisted .unnumbered} -  + The purpose of this chapter is to review methods of converting between raster and vector data formats, and to understand the circumstances in which this is useful. By way of example, this chapter focuses on topographic elevation and forest cover change in Colombia, but note that these are generic methods that can be applied in a wide variety of situations. @@ -352,86 +364,90 @@ The purpose of this chapter is to review methods of converting between raster an * Understanding raster and vector data in Earth Engine and their differing properties. * Knowing how and why to convert from raster to vector. * Knowing how and why to convert from vector to raster. -* Write a function and map it over a FeatureCollection. +* Write a function and map it over a FeatureCollection. ## Assumes you know how to:{.unlisted .unnumbered} * Import images and image collections, filter, and visualize (Part F1). -* Understand distinctions among Image, ImageCollection, Feature and FeatureCollection Earth Engine objects (Part F1, Part F2, Part F5). +* Understand distinctions among Image, ImageCollection, Feature and FeatureCollection Earth Engine objects (Part F1, Part F2, Part F5). * Perform basic image analysis: select bands, compute indices, create masks (Part F2). * Perform image morphological operations (Chap. F3.2). -* Understand the filter, map, reduce paradigm (Chap. F4.0). -* Write a function and map it over an ImageCollection (Chap. F4.0). -* Use reduceRegions to summarize an image in irregular shapes (Chap. F5.0). +* Understand the filter, map, reduce paradigm (Chap. F4.0). +* Write a function and map it over an ImageCollection (Chap. F4.0). +* Use reduceRegions to summarize an image in irregular shapes (Chap. F5.0). -::: +::: ## Introduction {.unlisted .unnumbered} -Raster data consists of regularly spaced pixels arranged into rows and columns, familiar as the format of satellite images. Vector data contains geometry features (i.e., points, lines, and polygons) describing locations and areas. Each data format has its advantages, and both will be encountered as part of GIS operations. +Raster data consists of regularly spaced pixels arranged into rows and columns, familiar as the format of satellite images. Vector data contains geometry features (i.e., points, lines, and polygons) describing locations and areas. Each data format has its advantages, and both will be encountered as part of GIS operations. Raster and vector data are commonly combined (e.g., extracting image information for a given location or clipping an image to an area of interest); however, there are also situations in which conversion between the two formats is useful. In making such conversions, it is important to consider the key advantages of each format. Rasters can store data efficiently where each pixel has a numerical value, while vector data can more effectively represent geometric features where homogenous areas have shared properties. Each format lends itself to distinctive analytical operations, and combining them can be powerful. In this exercise, we’ll use topographic elevation and forest change images in Colombia as well as a protected area feature collection to practice the conversion between raster and vector formats, and to identify situations in which this is worthwhile. -## Raster to Vector Conversion  +## Raster to Vector Conversion ### Raster to Polygons -In this section we will convert an elevation image (raster) to a feature collection (vector). We will start by loading the Global Multi-Resolution Terrain Elevation Data 2010 and the Global Administrative Unit Layers 2015 dataset to focus on Colombia. The elevation image is a raster at 7.5 arc-second spatial resolution containing a continuous measure of elevation in meters in each pixel. +In this section we will convert an elevation image (raster) to a feature collection (vector). We will start by loading the Global Multi-Resolution Terrain Elevation Data 2010 and the Global Administrative Unit Layers 2015 dataset to focus on Colombia. The elevation image is a raster at 7.5 arc-second spatial resolution containing a continuous measure of elevation in meters in each pixel. +```js // Load raster (elevation) and vector (colombia) datasets. -var elevation = ee.Image('USGS/GMTED2010').rename('elevation'); -var colombia = ee.FeatureCollection(       'FAO/GAUL_SIMPLIFIED_500m/2015/level0') -   .filter(ee.Filter.equals('ADM0_NAME', 'Colombia')); +var elevation = ee.Image('USGS/GMTED2010').rename('elevation'); +var colombia = ee.FeatureCollection( 'FAO/GAUL_SIMPLIFIED_500m/2015/level0') + .filter(ee.Filter.equals('ADM0_NAME', 'Colombia')); // Display elevation image. Map.centerObject(colombia, 7); Map.addLayer(elevation, { -   min: 0, -   max: 4000}, 'Elevation'); + min: 0, + max: 4000}, 'Elevation'); +``` When converting an image to a feature collection, we will aggregate the categorical elevation values into a set of categories to create polygon shapes of connected pixels with similar elevations. For this exercise, we will create four zones of elevation by grouping the altitudes to 0-100 m = 0, 100–200 m = 1, 200–500 m = 2, and >500 m = 3. +```js // Initialize image with zeros and define elevation zones. -var zones = ee.Image(0) -   .where(elevation.gt(100), 1) -   .where(elevation.gt(200), 2) -   .where(elevation.gt(500), 3); +var zones = ee.Image(0) + .where(elevation.gt(100), 1) + .where(elevation.gt(200), 2) + .where(elevation.gt(500), 3); // Mask pixels below sea level (<= 0 m) to retain only land areas. // Name the band with values 0-3 as 'zone'. zones = zones.updateMask(elevation.gt(0)).rename('zone'); Map.addLayer(zones, { -   min: 0, -   max: 3, -   palette: ['white', 'yellow', 'lime', 'green'], -   opacity: 0.7}, 'Elevation zones'); + min: 0, + max: 3, + palette: ['white', 'yellow', 'lime', 'green'], + opacity: 0.7}, 'Elevation zones'); -We will convert this zonal elevation image in Colombia to polygon shapes, which is a vector format (termed a FeatureCollection in Earth Engine), using the ee.Image.reduceToVectors method. This will create polygons delineating connected pixels with the same value. In doing so, we will use the same projection and spatial resolution as the image. Please note that loading the vectorized image in the native resolution (231.92 m) takes time to execute. For faster visualization, we set a coarse scale of 1,000 m. +``` +We will convert this zonal elevation image in Colombia to polygon shapes, which is a vector format (termed a FeatureCollection in Earth Engine), using the ee.Image.reduceToVectors method. This will create polygons delineating connected pixels with the same value. In doing so, we will use the same projection and spatial resolution as the image. Please note that loading the vectorized image in the native resolution (231.92 m) takes time to execute. For faster visualization, we set a coarse scale of 1,000 m. -var projection = elevation.projection(); -var scale = elevation.projection().nominalScale(); +var projection = elevation.projection(); +var scale = elevation.projection().nominalScale(); -var elevationVector = zones.reduceToVectors({ -   geometry: colombia.geometry(), -   crs: projection, -   scale: 1000, // scale   geometryType: 'polygon', -   eightConnected: false, -   labelProperty: 'zone', -   bestEffort: true, -   maxPixels: 1e13, -   tileScale: 3 // In case of error. +var elevationVector = zones.reduceToVectors({ + geometry: colombia.geometry(), + crs: projection, + scale: 1000, // scale geometryType: 'polygon', + eightConnected: false, + labelProperty: 'zone', + bestEffort: true, + maxPixels: 1e13, + tileScale: 3 // In case of error. }); print(elevationVector.limit(10)); -var elevationDrawn = elevationVector.draw({ -   color: 'black', -   strokeWidth: 1 +var elevationDrawn = elevationVector.draw({ + color: 'black', + strokeWidth: 1 }); Map.addLayer(elevationDrawn, {}, 'Elevation zone polygon'); @@ -441,47 +457,45 @@ Map.addLayer(elevationDrawn, {}, 'Elevation zone polygon'); ![](F5/image36.png) -![](F5/image7.png) +![Fig. F5.1.1 Raster-based elevation (top left) and zones (top right), vectorized elevation zones overlaid on the raster (bottom-left) and vectorized elevation zones only (bottom-right)](F5/image7.png) -Fig. F5.1.1 Raster-based elevation (top left) and zones (top right), vectorized elevation zones overlaid on the raster (bottom-left) and vectorized elevation zones only (bottom-right) -You may have realized that polygons consist of complex lines, including some small polygons with just one pixel. That happens when there are no surrounding pixels of the same elevation zone. You may not need a vector map with such details—if, for instance, you want to produce a regional or global map. We can use a morphological reducer focalMode to simplify the shape by defining a neighborhood size around a pixel. In this example, we will set the kernel radius as four pixels. This operation makes the resulting polygons look much smoother, but less precise (Fig. F5.1.2). +You may have realized that polygons consist of complex lines, including some small polygons with just one pixel. That happens when there are no surrounding pixels of the same elevation zone. You may not need a vector map with such details—if, for instance, you want to produce a regional or global map. We can use a morphological reducer focalMode to simplify the shape by defining a neighborhood size around a pixel. In this example, we will set the kernel radius as four pixels. This operation makes the resulting polygons look much smoother, but less precise (Fig. F5.1.2). -var zonesSmooth = zones.focalMode(4, 'square'); +var zonesSmooth = zones.focalMode(4, 'square'); zonesSmooth = zonesSmooth.reproject(projection.atScale(scale)); Map.addLayer(zonesSmooth, { -   min: 1, -   max: 3, -   palette: ['yellow', 'lime', 'green'], -   opacity: 0.7}, 'Elevation zones (smooth)'); + min: 1, + max: 3, + palette: ['yellow', 'lime', 'green'], + opacity: 0.7}, 'Elevation zones (smooth)'); -var elevationVectorSmooth = zonesSmooth.reduceToVectors({ -   geometry: colombia.geometry(), -   crs: projection, -   scale: scale, -   geometryType: 'polygon', -   eightConnected: false, -   labelProperty: 'zone', -   bestEffort: true, -   maxPixels: 1e13, -   tileScale: 3 +var elevationVectorSmooth = zonesSmooth.reduceToVectors({ + geometry: colombia.geometry(), + crs: projection, + scale: scale, + geometryType: 'polygon', + eightConnected: false, + labelProperty: 'zone', + bestEffort: true, + maxPixels: 1e13, + tileScale: 3 }); -var smoothDrawn = elevationVectorSmooth.draw({ -   color: 'black', -   strokeWidth: 1 +var smoothDrawn = elevationVectorSmooth.draw({ + color: 'black', + strokeWidth: 1 }); Map.addLayer(smoothDrawn, {}, 'Elevation zone polygon (smooth)'); -We can see now that the polygons have more distinct shapes with many fewer small polygons in the new map (Fig. F5.1.2). It is important to note that when you use methods like focalMode (or other, similar methods such as connectedComponents and connectedPixelCount), you need to reproject according to the original image in order to display properly with zoom using the interactive Code Editor. +We can see now that the polygons have more distinct shapes with many fewer small polygons in the new map (Fig. F5.1.2). It is important to note that when you use methods like focalMode (or other, similar methods such as connectedComponents and connectedPixelCount), you need to reproject according to the original image in order to display properly with zoom using the interactive Code Editor. ![](F5/image20.png) -![](F5/image37.png) +![Fig. F5.1.2 Before (left) and after (right) applying focalMode](F5/image37.png) -Fig. F5.1.2 Before (left) and after (right) applying focalMode ### Raster to Points @@ -489,86 +503,87 @@ Lastly, we will convert a small part of this elevation image into a point vector ![](F5/image24.png) -![](F5/image11.png) +![Fig. F5.1.3 Elevation point values with latitude and longitude](F5/image11.png) -Fig. F5.1.3 Elevation point values with latitude and longitude -The easiest way to do this is to use sample while activating the geometries parameter. This will extract the points at the centroid of the elevation pixel. +The easiest way to do this is to use sample while activating the geometries parameter. This will extract the points at the centroid of the elevation pixel. -var geometry = ee.Geometry.Polygon([ -   [-89.553, -0.929], -   [-89.436, -0.929], -   [-89.436, -0.866], -   [-89.553, -0.866], -   [-89.553, -0.929] +var geometry = ee.Geometry.Polygon([ + [-89.553, -0.929], + [-89.436, -0.929], + [-89.436, -0.866], + [-89.553, -0.866], + [-89.553, -0.929] ]); +```js // To zoom into the area, un-comment and run below // Map.centerObject(geometry,12); Map.addLayer(geometry, {}, 'Areas to extract points'); -var elevationSamples = elevation.sample({ -   region: geometry, -   projection: projection, -   scale: scale, -   geometries: true, +var elevationSamples = elevation.sample({ + region: geometry, + projection: projection, + scale: scale, + geometries: true, }); Map.addLayer(elevationSamples, {}, 'Points extracted'); // Add three properties to the output table: // 'Elevation', 'Longitude', and 'Latitude'. -elevationSamples = elevationSamples.map(function(feature) {   var geom = feature.geometry().coordinates();   return ee.Feature(null, {       'Elevation': ee.Number(feature.get(           'elevation')),       'Long': ee.Number(geom.get(0)),       'Lat': ee.Number(geom.get(1)) -   }); +elevationSamples = elevationSamples.map(function(feature) { var geom = feature.geometry().coordinates(); return ee.Feature(null, { 'Elevation': ee.Number(feature.get( 'elevation')), 'Long': ee.Number(geom.get(0)), 'Lat': ee.Number(geom.get(1)) + }); }); // Export as CSV. Export.table.toDrive({ -   collection: elevationSamples, -   description: 'extracted_points', -   fileFormat: 'CSV' + collection: elevationSamples, + description: 'extracted_points', + fileFormat: 'CSV' }); -We can also extract sample points per elevation zone. Below is an example of extracting 10 randomly selected points per elevation zone (Fig. F5.1.4). You can also set different values for each zone using classValues and classPoints parameters to modify the sampling intensity in each class. This may be useful, for instance, to generate point samples for a validation effort. +``` +We can also extract sample points per elevation zone. Below is an example of extracting 10 randomly selected points per elevation zone (Fig. F5.1.4). You can also set different values for each zone using classValues and classPoints parameters to modify the sampling intensity in each class. This may be useful, for instance, to generate point samples for a validation effort. -var elevationSamplesStratified = zones.stratifiedSample({ -   numPoints: 10, -   classBand: 'zone', -   region: geometry, -   scale: scale, -   projection: projection, -   geometries: true +var elevationSamplesStratified = zones.stratifiedSample({ + numPoints: 10, + classBand: 'zone', + region: geometry, + scale: scale, + projection: projection, + geometries: true }); Map.addLayer(elevationSamplesStratified, {}, 'Stratified samples'); -![](F5/image23.png) +![Fig. F5.1.4 Stratified sampling over different elevation zones](F5/image23.png) -Fig. F5.1.4 Stratified sampling over different elevation zones -::: {.callout-note} -Code Checkpoint F51a. The book’s repository contains a script that shows what your code should look like at this point. +:::{.callout-note} +Code Checkpoint F51a. The book’s repository contains a script that shows what your code should look like at this point. ::: ##3. A More Complex Example In this section we’ll use two global datasets, one to represent raster formats and the other vectors: -* The Global Forest Change (GFC) dataset: a raster dataset describing global tree cover and change for 2001–present. +* The Global Forest Change (GFC) dataset: a raster dataset describing global tree cover and change for 2001–present. * The World Protected Areas Database: a vector database of global protected areas. The objective will be to combine these two datasets to quantify rates of deforestation in protected areas in the “arc of deforestation” of the Colombian Amazon. The datasets can be loaded into Earth Engine with the following code: +```js // Read input data. // Note: these datasets are periodically updated. // Consider searching the Data Catalog for newer versions. -var gfc = ee.Image('UMD/hansen/global_forest_change_2020_v1_8'); -var wdpa = ee.FeatureCollection('WCMC/WDPA/current/polygons'); +var gfc = ee.Image('UMD/hansen/global_forest_change_2020_v1_8'); +var wdpa = ee.FeatureCollection('WCMC/WDPA/current/polygons'); // Print assets to show available layers and properties. print(gfc); print(wdpa.limit(10)); // Show first 10 records. -The GFC dataset (first presented in detail in Chap. F1.1) is a global set of rasters that quantify tree cover and change for the period beginning in 2001. We’ll use a single image from this dataset: +The GFC dataset (first presented in detail in Chap. F1.1) is a global set of rasters that quantify tree cover and change for the period beginning in 2001. We’ll use a single image from this dataset: * 'lossyear': a categorical raster of forest loss (1–20, corresponding to deforestation for the period 2001–2020), and 0 for no change @@ -577,125 +592,129 @@ The World Database on Protected Areas (WDPA) is a harmonized dataset of global t * 'NAME'’: the name of each protected area * ‘WDPA_PID’: a unique numerical ID for each protected area -To begin with, we’ll focus on forest change dynamics in ‘La Paya’, a small protected area in the Colombian Amazon. We’ll first visualize these data using the paint command, which is discussed in more detail in Chap. F5.3: +To begin with, we’ll focus on forest change dynamics in ‘La Paya’, a small protected area in the Colombian Amazon. We’ll first visualize these data using the paint command, which is discussed in more detail in Chap. F5.3: // Display deforestation. -var deforestation = gfc.select('lossyear'); +var deforestation = gfc.select('lossyear'); Map.addLayer(deforestation, { -   min: 1, -   max: 20, -   palette: ['yellow', 'orange', 'red'] + min: 1, + max: 20, + palette: ['yellow', 'orange', 'red'] }, 'Deforestation raster'); // Display WDPA data. -var protectedArea = wdpa.filter(ee.Filter.equals('NAME', 'La Paya')); +var protectedArea = wdpa.filter(ee.Filter.equals('NAME', 'La Paya')); // Display protected area as an outline (see F5.3 for paint()). -var protectedAreaOutline = ee.Image().byte().paint({ -   featureCollection: protectedArea, -   color: 1, -   width: 3 +var protectedAreaOutline = ee.Image().byte().paint({ + featureCollection: protectedArea, + color: 1, + width: 3 }); Map.addLayer(protectedAreaOutline, { -   palette: 'white'}, 'Protected area'); + palette: 'white'}, 'Protected area'); // Set up map display. Map.centerObject(protectedArea); Map.setOptions('SATELLITE'); +``` This will display the boundary of the La Paya protected area and deforestation in the region (Fig. F5.1.5). -![](F5/image55.png) +![Fig. F5.1.5 View of the La Paya protected area in the Colombian Amazon (in white), and deforestation over the period 2001–2020 (in yellows and reds, with darker colors indicating more recent changes)](F5/image55.png) -Fig. F5.1.5 View of the La Paya protected area in the Colombian Amazon (in white), and deforestation over the period 2001–2020 (in yellows and reds, with darker colors indicating more recent changes) -We can use Earth Engine to convert the deforestation raster to a set of polygons. The deforestation data are appropriate for this transformation as each deforestation event is labeled categorically by year, and change events are spatially contiguous. This is performed in Earth Engine using the ee.Image.reduceToVectors method, as described earlier in this section.   +We can use Earth Engine to convert the deforestation raster to a set of polygons. The deforestation data are appropriate for this transformation as each deforestation event is labeled categorically by year, and change events are spatially contiguous. This is performed in Earth Engine using the ee.Image.reduceToVectors method, as described earlier in this section. +```js // Convert from a deforestation raster to vector. -var deforestationVector = deforestation.reduceToVectors({ -   scale: deforestation.projection().nominalScale(), -   geometry: protectedArea.geometry(), -   labelProperty: 'lossyear', // Label polygons with a change year.   maxPixels: 1e13 +var deforestationVector = deforestation.reduceToVectors({ + scale: deforestation.projection().nominalScale(), + geometry: protectedArea.geometry(), + labelProperty: 'lossyear', // Label polygons with a change year. maxPixels: 1e13 }); // Count the number of individual change events print('Number of change events:', deforestationVector.size()); // Display deforestation polygons. Color outline by change year. -var deforestationVectorOutline = ee.Image().byte().paint({ -   featureCollection: deforestationVector, -   color: 'lossyear', -   width: 1 +var deforestationVectorOutline = ee.Image().byte().paint({ + featureCollection: deforestationVector, + color: 'lossyear', + width: 1 }); Map.addLayer(deforestationVectorOutline, { -   palette: ['yellow', 'orange', 'red'], -   min: 1, -   max: 20}, 'Deforestation vector'); + palette: ['yellow', 'orange', 'red'], + min: 1, + max: 20}, 'Deforestation vector'); +``` Fig. F5.1.6 shows a comparison of the raster versus vector representations of deforestation within the protected area. ![](F5/image42.png) -![](F5/image13.png) +![Fig. F5.1.6 Raster (left) versus vector (right) representations of deforestation data of the La Paya protected area](F5/image13.png) -Fig. F5.1.6 Raster (left) versus vector (right) representations of deforestation data of the La Paya protected area Having converted from raster to vector, a new set of operations becomes available for post-processing the deforestation data. We might, for instance, be interested in the number of individual change events each year (Fig. F5.1.7): -var chart = ui.Chart.feature -   .histogram({ -       features: deforestationVector, -       property: 'lossyear'   }) -   .setOptions({ -       hAxis: { -           title: 'Year'       }, -       vAxis: { -           title: 'Number of deforestation events'       }, -       legend: { -           position: 'none'       } -   });print(chart); +var chart = ui.Chart.feature + .histogram({ + features: deforestationVector, + property: 'lossyear' }) + .setOptions({ + hAxis: { + title: 'Year' }, + vAxis: { + title: 'Number of deforestation events' }, + legend: { + position: 'none' } + });print(chart); -![](F5/image15.png) +![Fig. F5.1.7 Plot of the number of deforestation events in La Paya for the years 2001–2020](F5/image15.png) -Fig. F5.1.7 Plot of the number of deforestation events in La Paya for the years 2001–2020 There might also be interest in generating point locations for individual change events (e.g., to aid a field campaign): +```js // Generate deforestation point locations. -var deforestationCentroids = deforestationVector.map(function(feat) {   return feat.centroid(); +var deforestationCentroids = deforestationVector.map(function(feat) { return feat.centroid(); }); Map.addLayer(deforestationCentroids, { -   color: 'darkblue'}, 'Deforestation centroids'); + color: 'darkblue'}, 'Deforestation centroids'); +``` The vector format allows for easy filtering to only deforestation events of interest, such as only the largest deforestation events: +```js // Add a new property to the deforestation FeatureCollection // describing the area of the change polygon. -deforestationVector = deforestationVector.map(function(feat) {   return feat.set('area', feat.geometry().area({ -       maxError: 10   }).divide(10000)); // Convert m^2 to hectare. +deforestationVector = deforestationVector.map(function(feat) { return feat.set('area', feat.geometry().area({ + maxError: 10 }).divide(10000)); // Convert m^2 to hectare. }); // Filter the deforestation FeatureCollection for only large-scale (>10 ha) changes -var deforestationLarge = deforestationVector.filter(ee.Filter.gt(   'area', 10)); +var deforestationLarge = deforestationVector.filter(ee.Filter.gt( 'area', 10)); // Display deforestation area outline by year. -var deforestationLargeOutline = ee.Image().byte().paint({ -   featureCollection: deforestationLarge, -   color: 'lossyear', -   width: 1 +var deforestationLargeOutline = ee.Image().byte().paint({ + featureCollection: deforestationLarge, + color: 'lossyear', + width: 1 }); Map.addLayer(deforestationLargeOutline, { -   palette: ['yellow', 'orange', 'red'], -   min: 1, -   max: 20}, 'Deforestation (>10 ha)'); + palette: ['yellow', 'orange', 'red'], + min: 1, + max: 20}, 'Deforestation (>10 ha)'); -::: {.callout-note} -Code Checkpoint F51b. The book’s repository contains a script that shows what your code should look like at this point. +``` +:::{.callout-note} +Code Checkpoint F51b. The book’s repository contains a script that shows what your code should look like at this point. ::: ### Raster Properties to Vector Fields @@ -703,201 +722,212 @@ Sometimes we want to extract information from a raster to be included in an exis The following script shows how this can be used to quantify a deforestation rate for a set of protected areas in the Colombian Amazon. +```js // Load required datasets. -var gfc = ee.Image('UMD/hansen/global_forest_change_2020_v1_8'); -var wdpa = ee.FeatureCollection('WCMC/WDPA/current/polygons'); +var gfc = ee.Image('UMD/hansen/global_forest_change_2020_v1_8'); +var wdpa = ee.FeatureCollection('WCMC/WDPA/current/polygons'); // Display deforestation. -var deforestation = gfc.select('lossyear'); +var deforestation = gfc.select('lossyear'); Map.addLayer(deforestation, { -   min: 1, -   max: 20, -   palette: ['yellow', 'orange', 'red'] + min: 1, + max: 20, + palette: ['yellow', 'orange', 'red'] }, 'Deforestation raster'); // Select protected areas in the Colombian Amazon. -var amazonianProtectedAreas = [   'Cordillera de los Picachos', 'La Paya', 'Nukak',   'Serrania de Chiribiquete',   'Sierra de la Macarena', 'Tinigua' +var amazonianProtectedAreas = [ 'Cordillera de los Picachos', 'La Paya', 'Nukak', 'Serrania de Chiribiquete', 'Sierra de la Macarena', 'Tinigua' ]; -var wdpaSubset = wdpa.filter(ee.Filter.inList('NAME', -   amazonianProtectedAreas)); +var wdpaSubset = wdpa.filter(ee.Filter.inList('NAME', + amazonianProtectedAreas)); // Display protected areas as an outline. -var protectedAreasOutline = ee.Image().byte().paint({ -   featureCollection: wdpaSubset, -   color: 1, -   width: 1 +var protectedAreasOutline = ee.Image().byte().paint({ + featureCollection: wdpaSubset, + color: 1, + width: 1 }); Map.addLayer(protectedAreasOutline, { -   palette: 'white'}, 'Amazonian protected areas'); + palette: 'white'}, 'Amazonian protected areas'); // Set up map display. Map.centerObject(wdpaSubset); Map.setOptions('SATELLITE'); -var scale = deforestation.projection().nominalScale(); +var scale = deforestation.projection().nominalScale(); // Use 'reduceRegions' to sum together pixel areas in each protected area. wdpaSubset = deforestation.gte(1) -   .multiply(ee.Image.pixelArea().divide(10000)).reduceRegions({ -       collection: wdpaSubset, -       reducer: ee.Reducer.sum().setOutputs([           'deforestation_area']), -       scale: scale -   }); + .multiply(ee.Image.pixelArea().divide(10000)).reduceRegions({ + collection: wdpaSubset, + reducer: ee.Reducer.sum().setOutputs([ 'deforestation_area']), + scale: scale + }); print(wdpaSubset); // Note the new 'deforestation_area' property. The output of this script is an estimate of deforested area in hectares for each reserve. However, as reserve sizes vary substantially by area, we can normalize by the total area of each reserve to quantify rates of change. // Normalize by area. -wdpaSubset = wdpaSubset.map(   function(feat) {       return feat.set('deforestation_rate',           ee.Number(feat.get('deforestation_area')) -           .divide(feat.area().divide(10000)) // m2 to ha           .divide(20) // number of years           .multiply(100)); // to percentage points   });// Print to identify rates of change per protected area. +wdpaSubset = wdpaSubset.map( function(feat) { return feat.set('deforestation_rate', ee.Number(feat.get('deforestation_area')) + .divide(feat.area().divide(10000)) // m2 to ha .divide(20) // number of years .multiply(100)); // to percentage points });// Print to identify rates of change per protected area. // Which has the fastest rate of loss? print(wdpaSubset.reduceColumns({ -   reducer: ee.Reducer.toList().repeat(2), -   selectors: ['NAME', 'deforestation_rate'] + reducer: ee.Reducer.toList().repeat(2), + selectors: ['NAME', 'deforestation_rate'] })); -::: {.callout-note} -Code Checkpoint F51c. The book’s repository contains a script that shows what your code should look like at this point. +``` +:::{.callout-note} +Code Checkpoint F51c. The book’s repository contains a script that shows what your code should look like at this point. ::: ## Vector-to-Raster Conversion -In Sect. 1, we used the protected area feature collection as its original vector format. In this section, we will rasterize the protected area polygons to produce a mask and use this to assess rates of forest change. +In Sect. 1, we used the protected area feature collection as its original vector format. In this section, we will rasterize the protected area polygons to produce a mask and use this to assess rates of forest change. ### Polygons to a Mask -The most common operation to convert from vector to raster is the production of binary image masks, describing whether a pixel intersects a line or falls within a polygon. To convert from vector to a raster mask, we can use the ee.FeatureCollection.reduceToImage method. Let’s continue with our example of the WDPA database and Global Forest Change data from the previous section: +The most common operation to convert from vector to raster is the production of binary image masks, describing whether a pixel intersects a line or falls within a polygon. To convert from vector to a raster mask, we can use the ee.FeatureCollection.reduceToImage method. Let’s continue with our example of the WDPA database and Global Forest Change data from the previous section: +```js // Load required datasets. -var gfc = ee.Image('UMD/hansen/global_forest_change_2020_v1_8'); -var wdpa = ee.FeatureCollection('WCMC/WDPA/current/polygons'); +var gfc = ee.Image('UMD/hansen/global_forest_change_2020_v1_8'); +var wdpa = ee.FeatureCollection('WCMC/WDPA/current/polygons'); // Get deforestation. -var deforestation = gfc.select('lossyear'); +var deforestation = gfc.select('lossyear'); // Generate a new property called 'protected' to apply to the output mask. -var wdpa = wdpa.map(function(feat) {   return feat.set('protected', 1); +var wdpa = wdpa.map(function(feat) { return feat.set('protected', 1); }); // Rasterize using the new property. // unmask() sets areas outside protected area polygons to 0. -var wdpaMask = wdpa.reduceToImage(['protected'], ee.Reducer.first()) -   .unmask(); +var wdpaMask = wdpa.reduceToImage(['protected'], ee.Reducer.first()) + .unmask(); // Center on Colombia. Map.setCenter(-75, 3, 6); // Display on map. Map.addLayer(wdpaMask, { -   min: 0, -   max: 1}, 'Protected areas (mask)'); + min: 0, + max: 1}, 'Protected areas (mask)'); +``` We can use this mask to, for example, highlight only deforestation that occurs within a protected area using logical operations: +```js // Set the deforestation layer to 0 where outside a protected area. -var deforestationProtected = deforestation.where(wdpaMask.eq(0), 0); +var deforestationProtected = deforestation.where(wdpaMask.eq(0), 0); // Update mask to hide where deforestation layer = 0 -var deforestationProtected = deforestationProtected -   .updateMask(deforestationProtected.gt(0)); +var deforestationProtected = deforestationProtected + .updateMask(deforestationProtected.gt(0)); // Display deforestation in protected areas Map.addLayer(deforestationProtected, { -   min: 1, -   max: 20, -   palette: ['yellow', 'orange', 'red'] + min: 1, + max: 20, + palette: ['yellow', 'orange', 'red'] }, 'Deforestation protected'); -In the above example we generated a simple binary mask, but reduceToImage can also preserve a numerical property of the input polygons. For example, we might want to be able to determine which protected area each pixel represents. In this case, we can produce an image with the unique ID of each protected area: +``` +In the above example we generated a simple binary mask, but reduceToImage can also preserve a numerical property of the input polygons. For example, we might want to be able to determine which protected area each pixel represents. In this case, we can produce an image with the unique ID of each protected area: +```js // Produce an image with unique ID of protected areas. -var wdpaId = wdpa.reduceToImage(['WDPAID'], ee.Reducer.first()); +var wdpaId = wdpa.reduceToImage(['WDPAID'], ee.Reducer.first()); Map.addLayer(wdpaId, { -   min: 1, -   max: 100000}, 'Protected area ID'); + min: 1, + max: 100000}, 'Protected area ID'); +``` This output can be useful when performing large-scale raster operations, such as efficiently calculating deforestation rates for multiple protected areas. -::: {.callout-note} -Code Checkpoint F51d. The book’s repository contains a script that shows what your code should look like at this point. +:::{.callout-note} +Code Checkpoint F51d. The book’s repository contains a script that shows what your code should look like at this point. ::: ### A More Complex Example -The reduceToImage method is not the only way to convert a feature collection to an image. We will create a distance image layer from the boundary of the protected area using distance. For this example, we return to the La Paya protected area explored in Sect. 1. +The reduceToImage method is not the only way to convert a feature collection to an image. We will create a distance image layer from the boundary of the protected area using distance. For this example, we return to the La Paya protected area explored in Sect. 1. +```js // Load required datasets. -var gfc = ee.Image('UMD/hansen/global_forest_change_2020_v1_8'); -var wdpa = ee.FeatureCollection('WCMC/WDPA/current/polygons'); +var gfc = ee.Image('UMD/hansen/global_forest_change_2020_v1_8'); +var wdpa = ee.FeatureCollection('WCMC/WDPA/current/polygons'); // Select a single protected area. -var protectedArea = wdpa.filter(ee.Filter.equals('NAME', 'La Paya')); +var protectedArea = wdpa.filter(ee.Filter.equals('NAME', 'La Paya')); // Maximum distance in meters is set in the brackets. -var distance = protectedArea.distance(1000000); +var distance = protectedArea.distance(1000000); Map.addLayer(distance, { -   min: 0, -   max: 20000, -   palette: ['white', 'grey', 'black'], -   opacity: 0.6}, 'Distance'); + min: 0, + max: 20000, + palette: ['white', 'grey', 'black'], + opacity: 0.6}, 'Distance'); Map.centerObject(protectedArea); +``` We can also show the distance inside and outside of the boundary by using the rasterized protected area (Fig. F5.1.8). +```js // Produce a raster of inside/outside the protected area. -var protectedAreaRaster = protectedArea.map(function(feat) {   return feat.set('protected', 1); +var protectedAreaRaster = protectedArea.map(function(feat) { return feat.set('protected', 1); }).reduceToImage(['protected'], ee.Reducer.first()); Map.addLayer(distance.updateMask(protectedAreaRaster), { -   min: 0, -   max: 20000}, 'Distance inside protected area'); + min: 0, + max: 20000}, 'Distance inside protected area'); Map.addLayer(distance.updateMask(protectedAreaRaster.unmask() .not()), { -   min: 0, -   max: 20000}, 'Distance outside protected area'); + min: 0, + max: 20000}, 'Distance outside protected area'); +``` ![](F5/image56.png) ![](F5/image9.png) -![](F5/image25.png) +![Fig. F5.1.8 Distance from the La Paya boundary (left), distance within the La Paya (middle), and distance outside the La Paya (right)](F5/image25.png) -Fig. F5.1.8 Distance from the La Paya boundary (left), distance within the La Paya (middle), and distance outside the La Paya (right) Sometimes it makes sense to work with objects in raster imagery. This is an unusual case of vector-like operations conducted with raster data. There is a good reason for this where the vector equivalent would be computationally burdensome. An example of this is estimating deforestation rates by distance to the edge of the protected area, as it is common that rates of change will be higher at the boundary of a protected area. We will create a distance raster with three zones from the La Paya boundary (>1 km, >2 km, >3 km, and >4 km) and to estimate the deforestation by distance from the boundary (Fig. F5.1.9). -var distanceZones = ee.Image(0) -   .where(distance.gt(0), 1) -   .where(distance.gt(1000), 2) -   .where(distance.gt(3000), 3) -   .updateMask(distance.lte(5000)); +var distanceZones = ee.Image(0) + .where(distance.gt(0), 1) + .where(distance.gt(1000), 2) + .where(distance.gt(3000), 3) + .updateMask(distance.lte(5000)); Map.addLayer(distanceZones, {}, 'Distance zones'); -var deforestation = gfc.select('loss'); -var deforestation1km = deforestation.updateMask(distanceZones.eq(1)); -var deforestation3km = deforestation.updateMask(distanceZones.lte(2)); -var deforestation5km = deforestation.updateMask(distanceZones.lte(3)); +var deforestation = gfc.select('loss'); +var deforestation1km = deforestation.updateMask(distanceZones.eq(1)); +var deforestation3km = deforestation.updateMask(distanceZones.lte(2)); +var deforestation5km = deforestation.updateMask(distanceZones.lte(3)); Map.addLayer(deforestation1km, { -   min: 0, -   max: 1}, 'Deforestation within a 1km buffer'); + min: 0, + max: 1}, 'Deforestation within a 1km buffer'); Map.addLayer(deforestation3km, { -   min: 0, -   max: 1, -   opacity: 0.5}, 'Deforestation within a 3km buffer'); + min: 0, + max: 1, + opacity: 0.5}, 'Deforestation within a 3km buffer'); Map.addLayer(deforestation5km, { -   min: 0, -   max: 1, -   opacity: 0.5}, 'Deforestation within a 5km buffer'); + min: 0, + max: 1, + opacity: 0.5}, 'Deforestation within a 5km buffer'); ![](F5/image22.png) @@ -905,38 +935,39 @@ Map.addLayer(deforestation5km, { ![](F5/image21.png) -![](F5/image26.png) +![Fig. F5.1.9 Distance zones (top left) and deforestation by zone (<1 km, <3 km, and <5 km)](F5/image26.png) -Fig. F5.1.9 Distance zones (top left) and deforestation by zone (<1 km, <3 km, and <5 km) Lastly, we can estimate the deforestation area within 1 km of the protected area but only outside of the boundary. -var deforestation1kmOutside = deforestation1km -   .updateMask(protectedAreaRaster.unmask().not()); +var deforestation1kmOutside = deforestation1km + .updateMask(protectedAreaRaster.unmask().not()); +```js // Get the value of each pixel in square meters // and divide by 10000 to convert to hectares. -var deforestation1kmOutsideArea = deforestation1kmOutside.eq(1) -   .multiply(ee.Image.pixelArea()).divide(10000); +var deforestation1kmOutsideArea = deforestation1kmOutside.eq(1) + .multiply(ee.Image.pixelArea()).divide(10000); // We need to set a larger geometry than the protected area // for the geometry parameter in reduceRegion(). -var deforestationEstimate = deforestation1kmOutsideArea -   .reduceRegion({ -       reducer: ee.Reducer.sum(), -       geometry: protectedArea.geometry().buffer(1000), -       scale: deforestation.projection().nominalScale() -   }); +var deforestationEstimate = deforestation1kmOutsideArea + .reduceRegion({ + reducer: ee.Reducer.sum(), + geometry: protectedArea.geometry().buffer(1000), + scale: deforestation.projection().nominalScale() + }); print('Deforestation within a 1km buffer outside the protected area (ha)', -   deforestationEstimate); + deforestationEstimate); -::: {.callout-note} -Code Checkpoint F51e. The book’s repository contains a script that shows what your code should look like at this point. +``` +:::{.callout-note} +Code Checkpoint F51e. The book’s repository contains a script that shows what your code should look like at this point. ::: ## Synthesis {.unnumbered} -Question 1. In this lab, we quantified rates of deforestation in La Paya. There is another protected area in the Colombian Amazon named Tinigua. By modifying the existing scripts, determine how the dynamics of forest change in Tinigua compare to those in La Paya with respect to: +Question 1. In this lab, we quantified rates of deforestation in La Paya. There is another protected area in the Colombian Amazon named Tinigua. By modifying the existing scripts, determine how the dynamics of forest change in Tinigua compare to those in La Paya with respect to: * the number of deforestation events * the year with the greatest number of change events @@ -947,7 +978,7 @@ Question 2. In Sect. 1.4, we only considered losses of tree cover, but many prot Question 3. In Sect. 2.2, we considered rates of deforestation in a buffer zone around La Paya. Estimate the deforestation rates inside of La Paya using buffer zones. Is forest loss more common close to the boundary of the reserve? -Question 4. Sometimes it’s advantageous to perform processing using raster operations, particularly at large scales. It is possible to perform many of the tasks in Sect. 1.3 and 1.4 by first converting the protected area vector to raster, and then using only raster operations. As an example, can you display only deforestation events >10 ha in La Paya using only raster data? (Hint: Consider using ee.Image.connectedPixelCount. You may also want to also look at Sect. 2.1). +Question 4. Sometimes it’s advantageous to perform processing using raster operations, particularly at large scales. It is possible to perform many of the tasks in Sect. 1.3 and 1.4 by first converting the protected area vector to raster, and then using only raster operations. As an example, can you display only deforestation events >10 ha in La Paya using only raster data? (Hint: Consider using ee.Image.connectedPixelCount. You may also want to also look at Sect. 2.1). ## Conclusion {.unnumbered} @@ -961,12 +992,12 @@ In this chapter, you learned how to convert raster to vector and vice versa. Mor -::: {.callout-tip} +:::{.callout-tip} # Chapter Information ## Author {.unlisted .unnumbered} -  + Sara Winsemius and Justin Braaten @@ -978,7 +1009,7 @@ Sara Winsemius and Justin Braaten The purpose of this chapter is to extract values from rasters for intersecting points or polygons. We will lay out the process and a function to calculate zonal statistics, which includes optional parameters to modify the function, and then apply the process to three examples using different raster datasets and combinations of parameters. ## Learning Outcomes {.unlisted .unnumbered} -  + * Buffering points as square or circular regions. * Writing and applying functions with optional parameters. @@ -990,19 +1021,19 @@ The purpose of this chapter is to extract values from rasters for intersecting p * Recognize similarities and differences among Landsat 5, 7, and 8 spectral bands (Part F1, Part F2, Part F3). -* Understand distinctions among Image, ImageCollection, Feature and FeatureCollection Earth Engine objects (Part F1, Part F2, Part F5). +* Understand distinctions among Image, ImageCollection, Feature and FeatureCollection Earth Engine objects (Part F1, Part F2, Part F5). * Use drawing tools to create points, lines, and polygons (Chap. F2.1). -* Write a function and map it over an ImageCollection (Chap. F4.0). +* Write a function and map it over an ImageCollection (Chap. F4.0). * Mask cloud, cloud shadow, snow/ice, and other undesired pixels (Chap. F4.3). * Export calculated data to tables with Tasks (Chap. F5.0). -* Understand the differences between raster and vector data (Chap. F5.0, Chap. F5.1). -* Write a function and map it over a FeatureCollection (Chap. F5.1). +* Understand the differences between raster and vector data (Chap. F5.0, Chap. F5.1). +* Write a function and map it over a FeatureCollection (Chap. F5.1). -## Introduction to Theory  +## Introduction to Theory -Anyone working with field data collected at plots will likely need to summarize raster-based data associated with those plots. For instance, they need to know the Normalized Difference Vegetation Index (NDVI), precipitation, or elevation for each plot (or surrounding region). Calculating statistics from a raster within given regions is called zonal statistics. Zonal statistics were calculated in Chaps. F5.0 and F5.1 using ee.Image.ReduceRegions. Here, we present a more general approach to calculating zonal statistics with a custom function that works for both ee.Image and ee.ImageCollection objects. In addition to its flexibility, the reduction method used here is less prone to “Computed value is too large” errors that can occur when using ReduceRegions with very large or complex ee.FeatureCollection object inputs. +Anyone working with field data collected at plots will likely need to summarize raster-based data associated with those plots. For instance, they need to know the Normalized Difference Vegetation Index (NDVI), precipitation, or elevation for each plot (or surrounding region). Calculating statistics from a raster within given regions is called zonal statistics. Zonal statistics were calculated in Chaps. F5.0 and F5.1 using ee.Image.ReduceRegions. Here, we present a more general approach to calculating zonal statistics with a custom function that works for both ee.Image and ee.ImageCollection objects. In addition to its flexibility, the reduction method used here is less prone to “Computed value is too large” errors that can occur when using ReduceRegions with very large or complex ee.FeatureCollection object inputs. -The zonal statistics function in this chapter works for an Image or an ImageCollection. Running the function over an ImageCollection will produce a table with values from each image in the collection per point. Image collections can be processed before extraction as needed—for example, by masking clouds from satellite imagery or by constraining the dates needed for a particular research question. In this tutorial, the data extracted from rasters are exported to a table for analysis, where each row of the table corresponds to a unique point-image combination. +The zonal statistics function in this chapter works for an Image or an ImageCollection. Running the function over an ImageCollection will produce a table with values from each image in the collection per point. Image collections can be processed before extraction as needed—for example, by masking clouds from satellite imagery or by constraining the dates needed for a particular research question. In this tutorial, the data extracted from rasters are exported to a table for analysis, where each row of the table corresponds to a unique point-image combination. In fieldwork, researchers often work with plots, which are commonly recorded as polygon files or as a center point with a set radius. It is rare that plots will be set directly in the center of pixels from your desired raster dataset, and many field GPS units have positioning errors. Because of these issues, it may be important to use a statistic of adjacent pixels (as described in Chap. F3.2) to estimate the central value in what’s often called a neighborhood mean or focal mean (Cansler and McKenzie 2012, Miller and Thode 2007). @@ -1020,7 +1051,7 @@ Two functions are provided; copy and paste them into your script: Our first function, bufferPoints, returns a function for adding a buffer to points and optionally transforming to rectangular bounds (see Table F5.2.1). -Table F5.2.1 Parameters for bufferPoints +Table F5.2.1 Parameters for bufferPoints Parameter @@ -1040,19 +1071,19 @@ Boolean An optional flag indicating whether to transform buffered point (i.e., a circle) to square bounds. -function bufferPoints(radius, bounds) {   return function(pt) { -       pt = ee.Feature(pt);       return bounds ? pt.buffer(radius).bounds() : pt.buffer( -           radius); -   }; +function bufferPoints(radius, bounds) { return function(pt) { + pt = ee.Feature(pt); return bounds ? pt.buffer(radius).bounds() : pt.buffer( + radius); + }; } ### Function: zonalStats(fc, params) -The second function, zonalStats, reduces images in an ImageCollection by regions defined in a FeatureCollection. Note that reductions can return null statistics that you might want to filter out of the resulting feature collection. Null statistics occur when there are no valid pixels intersecting the region being reduced. This situation can be caused by points that are outside of an image or in regions that are masked for quality or clouds. +The second function, zonalStats, reduces images in an ImageCollection by regions defined in a FeatureCollection. Note that reductions can return null statistics that you might want to filter out of the resulting feature collection. Null statistics occur when there are no valid pixels intersecting the region being reduced. This situation can be caused by points that are outside of an image or in regions that are masked for quality or clouds. This function is written to include many optional parameters (see Table F5.2.2). Look at the function carefully and note how it is written to include defaults that make it easy to apply the basic function while allowing customization. -Table F5.2.2 Parameters for zonalStats +Table F5.2.2 Parameters for zonalStats Parameter @@ -1130,46 +1161,46 @@ The desired name of the datetime field. The datetime refers to the 'system:time_ String -The desired datetime format. Use ISO 8601 data string standards. The datetime string is derived from the 'system:time_start' value of the ee.Image being reduced. Optional. +The desired datetime format. Use ISO 8601 data string standards. The datetime string is derived from the 'system:time_start' value of the ee.Image being reduced. Optional. -function zonalStats(ic, fc, params) {   // Initialize internal params dictionary.   var _params = { -       reducer: ee.Reducer.mean(), -       scale: null, -       crs: null, -       bands: null, -       bandsRename: null, -       imgProps: null, -       imgPropsRename: null, -       datetimeName: 'datetime', -       datetimeFormat: 'YYYY-MM-dd HH:mm:ss'   };   // Replace initialized params with provided params.   if (params) {       for (var param in params) { -           _params[param] = params[param] || _params[param]; -       } -   }   // Set default parameters based on an image representative.   var imgRep = ic.first();   var nonSystemImgProps = ee.Feature(null) -       .copyProperties(imgRep).propertyNames();   if (!_params.bands) _params.bands = imgRep.bandNames();   if (!_params.bandsRename) _params.bandsRename = _params.bands;   if (!_params.imgProps) _params.imgProps = nonSystemImgProps;   if (!_params.imgPropsRename) _params.imgPropsRename = _params -       .imgProps;   // Map the reduceRegions function over the image collection.   var results = ic.map(function(img) {       // Select bands (optionally rename), set a datetime & timestamp property.       img = ee.Image(img.select(_params.bands, _params -               .bandsRename))           // Add datetime and timestamp features.           .set(_params.datetimeName, img.date().format( -               _params.datetimeFormat))           .set('timestamp', img.get('system:time_start'));       // Define final image property dictionary to set in output features.       var propsFrom = ee.List(_params.imgProps)           .cat(ee.List([_params.datetimeName,           'timestamp']));       var propsTo = ee.List(_params.imgPropsRename)           .cat(ee.List([_params.datetimeName,           'timestamp']));       var imgProps = img.toDictionary(propsFrom).rename( -           propsFrom, propsTo);       // Subset points that intersect the given image.       var fcSub = fc.filterBounds(img.geometry());       // Reduce the image by regions.       return img.reduceRegions({ -               collection: fcSub, -               reducer: _params.reducer,               scale: _params.scale,               crs: _params.crs -           })           // Add metadata to each feature.           .map(function(f) {               return f.set(imgProps); -           });       // Converts the feature collection of feature collections to a single       //feature collection.   }).flatten();   return results; +function zonalStats(ic, fc, params) { // Initialize internal params dictionary. var _params = { + reducer: ee.Reducer.mean(), + scale: null, + crs: null, + bands: null, + bandsRename: null, + imgProps: null, + imgPropsRename: null, + datetimeName: 'datetime', + datetimeFormat: 'YYYY-MM-dd HH:mm:ss' }; // Replace initialized params with provided params. if (params) { for (var param in params) { + _params[param] = params[param] || _params[param]; + } + } // Set default parameters based on an image representative. var imgRep = ic.first(); var nonSystemImgProps = ee.Feature(null) + .copyProperties(imgRep).propertyNames(); if (!_params.bands) _params.bands = imgRep.bandNames(); if (!_params.bandsRename) _params.bandsRename = _params.bands; if (!_params.imgProps) _params.imgProps = nonSystemImgProps; if (!_params.imgPropsRename) _params.imgPropsRename = _params + .imgProps; // Map the reduceRegions function over the image collection. var results = ic.map(function(img) { // Select bands (optionally rename), set a datetime & timestamp property. img = ee.Image(img.select(_params.bands, _params + .bandsRename)) // Add datetime and timestamp features. .set(_params.datetimeName, img.date().format( + _params.datetimeFormat)) .set('timestamp', img.get('system:time_start')); // Define final image property dictionary to set in output features. var propsFrom = ee.List(_params.imgProps) .cat(ee.List([_params.datetimeName, 'timestamp'])); var propsTo = ee.List(_params.imgPropsRename) .cat(ee.List([_params.datetimeName, 'timestamp'])); var imgProps = img.toDictionary(propsFrom).rename( + propsFrom, propsTo); // Subset points that intersect the given image. var fcSub = fc.filterBounds(img.geometry()); // Reduce the image by regions. return img.reduceRegions({ + collection: fcSub, + reducer: _params.reducer, scale: _params.scale, crs: _params.crs + }) // Add metadata to each feature. .map(function(f) { return f.set(imgProps); + }); // Converts the feature collection of feature collections to a single //feature collection. }).flatten(); return results; } ## Point Collection Creation -Below, we create a set of points that form the basis of the zonal statistics calculations. Note that a unique plot_id property is added to each point. A unique plot or point ID is important to include in your vector dataset for future filtering and joining. +Below, we create a set of points that form the basis of the zonal statistics calculations. Note that a unique plot_id property is added to each point. A unique plot or point ID is important to include in your vector dataset for future filtering and joining. -var pts = ee.FeatureCollection([   ee.Feature(ee.Geometry.Point([-118.6010, 37.0777]), { -       plot_id: 1   }),   ee.Feature(ee.Geometry.Point([-118.5896, 37.0778]), { -       plot_id: 2   }),   ee.Feature(ee.Geometry.Point([-118.5842, 37.0805]), { -       plot_id: 3   }),   ee.Feature(ee.Geometry.Point([-118.5994, 37.0936]), { -       plot_id: 4   }),   ee.Feature(ee.Geometry.Point([-118.5861, 37.0567]), { -       plot_id: 5   }) +var pts = ee.FeatureCollection([ ee.Feature(ee.Geometry.Point([-118.6010, 37.0777]), { + plot_id: 1 }), ee.Feature(ee.Geometry.Point([-118.5896, 37.0778]), { + plot_id: 2 }), ee.Feature(ee.Geometry.Point([-118.5842, 37.0805]), { + plot_id: 3 }), ee.Feature(ee.Geometry.Point([-118.5994, 37.0936]), { + plot_id: 4 }), ee.Feature(ee.Geometry.Point([-118.5861, 37.0567]), { + plot_id: 5 }) ]);print('Points of interest', pts); -::: {.callout-note} -Code Checkpoint F52a. The book’s repository contains a script that shows what your code should look like at this point. +:::{.callout-note} +Code Checkpoint F52a. The book’s repository contains a script that shows what your code should look like at this point. ::: ## Neighborhood Statistic Examples @@ -1187,68 +1218,72 @@ This example demonstrates how to calculate zonal statistics for a single multiba ###Buffer the Points -Nex, we will apply a 45 m radius buffer to the points defined previously by mapping the bufferPoints function over the feature collection. The radius is set to 45 m to correspond to the 90 m pixel resolution of the DEM. In this case, circles are used instead of squares (set the second argument as false, i.e., do not use bounds). +Nex, we will apply a 45 m radius buffer to the points defined previously by mapping the bufferPoints function over the feature collection. The radius is set to 45 m to correspond to the 90 m pixel resolution of the DEM. In this case, circles are used instead of squares (set the second argument as false, i.e., do not use bounds). +```js // Buffer the points. -var ptsTopo = pts.map(bufferPoints(45, false)); +var ptsTopo = pts.map(bufferPoints(45, false)); +``` ###Calculate Zonal Statistics -There are two important things to note about the zonalStats function that this example addresses: +There are two important things to note about the zonalStats function that this example addresses: * It accepts only an ee.ImageCollection, not an ee.Image; single images must be wrapped in an ImageCollection. -* It expects every image in the input image collection to have a timestamp property named 'system:time_start' with values representing milliseconds from 00:00:00 UTC on 1 January 1970. Most datasets should have this property, if not, one should be added. +* It expects every image in the input image collection to have a timestamp property named 'system:time_start' with values representing milliseconds from 00:00:00 UTC on 1 January 1970. Most datasets should have this property, if not, one should be added. +```js // Import the MERIT global elevation dataset. -var elev = ee.Image('MERIT/DEM/v1_0_3'); +var elev = ee.Image('MERIT/DEM/v1_0_3'); // Calculate slope from the DEM. -var slope = ee.Terrain.slope(elev); +var slope = ee.Terrain.slope(elev); // Concatenate elevation and slope as two bands of an image. -var topo = ee.Image.cat(elev, slope) -    // Computed images do not have a 'system:time_start' property; add one based -    // on when the data were collected.   .set('system:time_start', ee.Date('2000-01-01').millis()); +var topo = ee.Image.cat(elev, slope) + // Computed images do not have a 'system:time_start' property; add one based + // on when the data were collected. .set('system:time_start', ee.Date('2000-01-01').millis()); // Wrap the single image in an ImageCollection for use in the // zonalStats function. -var topoCol = ee.ImageCollection([topo]); +var topoCol = ee.ImageCollection([topo]); -Define arguments for the zonalStats function and then run it. Note that we are accepting defaults for the reducer, scale, Coordinate Reference System (CRS), and image properties to copy over to the resulting feature collection. Refer to the function definition above for defaults. +``` +Define arguments for the zonalStats function and then run it. Note that we are accepting defaults for the reducer, scale, Coordinate Reference System (CRS), and image properties to copy over to the resulting feature collection. Refer to the function definition above for defaults. +```js // Define parameters for the zonalStats function. -var params = { -   bands: [0, 1], -   bandsRename: ['elevation', 'slope'] +var params = { + bands: [0, 1], + bandsRename: ['elevation', 'slope'] }; // Extract zonal statistics per point per image. -var ptsTopoStats = zonalStats(topoCol, ptsTopo, params);print('Topo zonal stats table', ptsTopoStats); +var ptsTopoStats = zonalStats(topoCol, ptsTopo, params);print('Topo zonal stats table', ptsTopoStats); // Display the layers on the map. Map.setCenter(-118.5957, 37.0775, 13); Map.addLayer(topoCol.select(0), { -   min: 2400, -   max: 4200}, 'Elevation'); + min: 2400, + max: 4200}, 'Elevation'); Map.addLayer(topoCol.select(1), { -   min: 0, -   max: 60}, 'Slope'); + min: 0, + max: 60}, 'Slope'); Map.addLayer(pts, { -   color: 'purple'}, 'Points'); + color: 'purple'}, 'Points'); Map.addLayer(ptsTopo, { -   color: 'yellow'}, 'Points w/ buffer'); + color: 'yellow'}, 'Points w/ buffer'); -The result is a copy of the buffered point feature collection with new properties added for the region reduction of each selected image band according to the given reducer. A part of the FeatureCollection is shown in Fig. F5.2.1. The data in that FeatureCollection corresponds to a table containing the information of Table F5.2.3. See Fig. F5.2.2 for a graphical representation of the points and the topographic data being summarized. +``` +The result is a copy of the buffered point feature collection with new properties added for the region reduction of each selected image band according to the given reducer. A part of the FeatureCollection is shown in Fig. F5.2.1. The data in that FeatureCollection corresponds to a table containing the information of Table F5.2.3. See Fig. F5.2.2 for a graphical representation of the points and the topographic data being summarized. -![](F5/image29.png) +![Fig. F5.2.1 A part of the FeatureCollection produced by calculating the zonal statistics](F5/image29.png) -Fig. F5.2.1 A part of the FeatureCollection produced by calculating the zonal statistics -![](F5/image5.png) +![Fig. F5.2.2 Sample points and topographic slope. Elevation and slope values for regions intersecting each buffered point are reduced and attached as properties of the points.](F5/image5.png) -Fig. F5.2.2 Sample points and topographic slope. Elevation and slope values for regions intersecting each buffered point are reduced and attached as properties of the points. -Table F5.2.3 Example output from zonalStats organized as a table. Rows correspond to collection features and columns are feature properties. Note that elevation and slope values in this table are rounded to the nearest tenth for brevity. +Table F5.2.3 Example output from zonalStats organized as a table. Rows correspond to collection features and columns are feature properties. Note that elevation and slope values in this table are rounded to the nearest tenth for brevity. plot_id @@ -1312,283 +1347,296 @@ slope ### MODIS Time Series -A time series of MODIS eight-day surface reflectance composites demonstrates how to calculate zonal statistics for a multiband ImageCollection that requires no preprocessing, such as cloud masking or computation. Note that there is no built-in function for performing region reductions on ImageCollection objects. The zonalStats function that we are using for reduction is mapping the reduceRegions function over an ImageCollection. +A time series of MODIS eight-day surface reflectance composites demonstrates how to calculate zonal statistics for a multiband ImageCollection that requires no preprocessing, such as cloud masking or computation. Note that there is no built-in function for performing region reductions on ImageCollection objects. The zonalStats function that we are using for reduction is mapping the reduceRegions function over an ImageCollection. ###Buffer the Points -In this example, suppose the point collection represents center points for field plots that are 100 m x 100 m, and apply a 50 m radius buffer to the points to match the size of the plot. Since we want zonal statistics for square plots, set the second argument of the bufferPoints function to true, so that the bounds of the buffered points are returned. +In this example, suppose the point collection represents center points for field plots that are 100 m x 100 m, and apply a 50 m radius buffer to the points to match the size of the plot. Since we want zonal statistics for square plots, set the second argument of the bufferPoints function to true, so that the bounds of the buffered points are returned. -var ptsModis = pts.map(bufferPoints(50, true)); +var ptsModis = pts.map(bufferPoints(50, true)); ###Calculate Zonal Statistic Import the MODIS 500 m global eight-day surface reflectance composite collection and filter the collection to include data for July, August, and September from 2015 through 2019. -var modisCol = ee.ImageCollection('MODIS/006/MOD09A1') -   .filterDate('2015-01-01', '2020-01-01') -   .filter(ee.Filter.calendarRange(183, 245, 'DAY_OF_YEAR')); +var modisCol = ee.ImageCollection('MODIS/006/MOD09A1') + .filterDate('2015-01-01', '2020-01-01') + .filter(ee.Filter.calendarRange(183, 245, 'DAY_OF_YEAR')); Reduce each image in the collection by each plot according to the following parameters. Note that this time the reducer is defined as the neighborhood median (ee.Reducer.median) instead of the default mean, and that scale, CRS, and properties for the datetime are explicitly defined. +```js // Define parameters for the zonalStats function. -var params = { -   reducer: ee.Reducer.median(), -   scale: 500, -   crs: 'EPSG:5070', -   bands: ['sur_refl_b01', 'sur_refl_b02', 'sur_refl_b06'], -   bandsRename: ['modis_red', 'modis_nir', 'modis_swir'], -   datetimeName: 'date', -   datetimeFormat: 'YYYY-MM-dd' +var params = { + reducer: ee.Reducer.median(), + scale: 500, + crs: 'EPSG:5070', + bands: ['sur_refl_b01', 'sur_refl_b02', 'sur_refl_b06'], + bandsRename: ['modis_red', 'modis_nir', 'modis_swir'], + datetimeName: 'date', + datetimeFormat: 'YYYY-MM-dd' }; // Extract zonal statistics per point per image. -var ptsModisStats = zonalStats(modisCol, ptsModis, params);print('Limited MODIS zonal stats table', ptsModisStats.limit(50)); +var ptsModisStats = zonalStats(modisCol, ptsModis, params);print('Limited MODIS zonal stats table', ptsModisStats.limit(50)); +``` The result is a feature collection with a feature for all combinations of plots and images. Interpreted as a table, the result has 200 rows (5 plots times 40 images) and as many columns as there are feature properties. Feature properties include those from the plot asset and the image, and any associated non-system image properties. Note that the printed results are limited to the first 50 features for brevity. ### Landsat Time Series This example combines Landsat surface reflectance imagery across three instruments: Thematic Mapper (TM) from Landsat 5, Enhanced Thematic Mapper Plus (ETM+) from Landsat 7, and Operational Land Imager (OLI) from Landsat 8. -The following section prepares these collections so that band names are consistent and cloud masks are applied. Reflectance among corresponding bands are roughly congruent for the three sensors when using the surface reflectance product; therefore the processing steps that follow do not address inter-sensor harmonization. Review the current literature on inter-sensor harmonization practices if you'd like to apply a correction. +The following section prepares these collections so that band names are consistent and cloud masks are applied. Reflectance among corresponding bands are roughly congruent for the three sensors when using the surface reflectance product; therefore the processing steps that follow do not address inter-sensor harmonization. Review the current literature on inter-sensor harmonization practices if you'd like to apply a correction. ###Prepare the Landsat Image Collection -First, define the function to mask cloud and shadow pixels (See Chap. F4.3 for more detail on cloud masking). +First, define the function to mask cloud and shadow pixels (See Chap. F4.3 for more detail on cloud masking). +```js // Mask clouds from images and apply scaling factors. -function maskScale(img) {   var qaMask = img.select('QA_PIXEL').bitwiseAnd(parseInt('11111',       2)).eq(0);   var saturationMask = img.select('QA_RADSAT').eq(0);   // Apply the scaling factors to the appropriate bands.   var getFactorImg = function(factorNames) {       var factorList = img.toDictionary().select(factorNames) -           .values();       return ee.Image.constant(factorList); -   };   var scaleImg = getFactorImg(['REFLECTANCE_MULT_BAND_.']);   var offsetImg = getFactorImg(['REFLECTANCE_ADD_BAND_.']);   var scaled = img.select('SR_B.').multiply(scaleImg).add( -   offsetImg);   // Replace the original bands with the scaled ones and apply the masks.   return img.addBands(scaled, null, true) -       .updateMask(qaMask) -       .updateMask(saturationMask); +function maskScale(img) { var qaMask = img.select('QA_PIXEL').bitwiseAnd(parseInt('11111', 2)).eq(0); var saturationMask = img.select('QA_RADSAT').eq(0); // Apply the scaling factors to the appropriate bands. var getFactorImg = function(factorNames) { var factorList = img.toDictionary().select(factorNames) + .values(); return ee.Image.constant(factorList); + }; var scaleImg = getFactorImg(['REFLECTANCE_MULT_BAND_.']); var offsetImg = getFactorImg(['REFLECTANCE_ADD_BAND_.']); var scaled = img.select('SR_B.').multiply(scaleImg).add( + offsetImg); // Replace the original bands with the scaled ones and apply the masks. return img.addBands(scaled, null, true) + .updateMask(qaMask) + .updateMask(saturationMask); } Next, define functions to select and rename the bands of interest for the Operational Land Imager (OLI) aboard Landsat 8, and for the TM/ETM+ imagers aboard earlier Landsats. This is important because the band numbers are different for OLI and TM/ETM+, and it will make future index calculations easier. // Selects and renames bands of interest for Landsat OLI. -function renameOli(img) {   return img.select( -       ['SR_B2', 'SR_B3', 'SR_B4', 'SR_B5', 'SR_B6', 'SR_B7'], -       ['Blue', 'Green', 'Red', 'NIR', 'SWIR1', 'SWIR2']); +function renameOli(img) { return img.select( + ['SR_B2', 'SR_B3', 'SR_B4', 'SR_B5', 'SR_B6', 'SR_B7'], + ['Blue', 'Green', 'Red', 'NIR', 'SWIR1', 'SWIR2']); } // Selects and renames bands of interest for TM/ETM+. -function renameEtm(img) {   return img.select( -       ['SR_B1', 'SR_B2', 'SR_B3', 'SR_B4', 'SR_B5', 'SR_B7'], -       ['Blue', 'Green', 'Red', 'NIR', 'SWIR1', 'SWIR2']); +function renameEtm(img) { return img.select( + ['SR_B1', 'SR_B2', 'SR_B3', 'SR_B4', 'SR_B5', 'SR_B7'], + ['Blue', 'Green', 'Red', 'NIR', 'SWIR1', 'SWIR2']); } Combine the cloud mask and band renaming functions into preparation functions for OLI and TM/ETM+. Add any other sensor-specific preprocessing steps that you’d like to the functions below. // Prepares (cloud masks and renames) OLI images. -function prepOli(img) { -   img = maskScale(img); -   img = renameOli(img);   return img; +function prepOli(img) { + img = maskScale(img); + img = renameOli(img); return img; }// Prepares (cloud masks and renames) TM/ETM+ images. -function prepEtm(img) { -   img = maskScale(img); -   img = renameEtm(img);   return img; +function prepEtm(img) { + img = maskScale(img); + img = renameEtm(img); return img; } Get the Landsat surface reflectance collections for OLI, ETM+, and TM sensors. Filter them by the bounds of the point feature collection and apply the relevant image preparation function. -var ptsLandsat = pts.map(bufferPoints(15, true)); +var ptsLandsat = pts.map(bufferPoints(15, true)); -var oliCol = ee.ImageCollection('LANDSAT/LC08/C02/T1_L2') -   .filterBounds(ptsLandsat) -   .map(prepOli); +var oliCol = ee.ImageCollection('LANDSAT/LC08/C02/T1_L2') + .filterBounds(ptsLandsat) + .map(prepOli); -var etmCol = ee.ImageCollection('LANDSAT/LE07/C02/T1_L2') -   .filterBounds(ptsLandsat) -   .map(prepEtm); +var etmCol = ee.ImageCollection('LANDSAT/LE07/C02/T1_L2') + .filterBounds(ptsLandsat) + .map(prepEtm); -var tmCol = ee.ImageCollection('LANDSAT/LT05/C02/T1_L2') -   .filterBounds(ptsLandsat) -   .map(prepEtm); +var tmCol = ee.ImageCollection('LANDSAT/LT05/C02/T1_L2') + .filterBounds(ptsLandsat) + .map(prepEtm); +``` Merge the prepared sensor collections. -var landsatCol = oliCol.merge(etmCol).merge(tmCol); +var landsatCol = oliCol.merge(etmCol).merge(tmCol); ###Calculate Zonal Statistics -Reduce each image in the collection by each plot according to the following parameters. Note that this example defines the imgProps and imgPropsRename parameters to copy over and rename just two selected image properties: Landsat image ID and the satellite that collected the data. It also uses the max reducer, which, as an unweighted reducer, will return the maximum value from pixels that have their centroid within the buffer (see Sect. 4.1 below for more details). +Reduce each image in the collection by each plot according to the following parameters. Note that this example defines the imgProps and imgPropsRename parameters to copy over and rename just two selected image properties: Landsat image ID and the satellite that collected the data. It also uses the max reducer, which, as an unweighted reducer, will return the maximum value from pixels that have their centroid within the buffer (see Sect. 4.1 below for more details). +```js // Define parameters for the zonalStats function. -var params = { -   reducer: ee.Reducer.max(), -   scale: 30, -   crs: 'EPSG:5070', -   bands: ['Blue', 'Green', 'Red', 'NIR', 'SWIR1', 'SWIR2'], -   bandsRename: ['ls_blue', 'ls_green', 'ls_red', 'ls_nir',       'ls_swir1', 'ls_swir2'   ], -   imgProps: ['SENSOR_ID', 'SPACECRAFT_ID'], -   imgPropsRename: ['img_id', 'satellite'], -   datetimeName: 'date', -   datetimeFormat: 'YYYY-MM-dd' +var params = { + reducer: ee.Reducer.max(), + scale: 30, + crs: 'EPSG:5070', + bands: ['Blue', 'Green', 'Red', 'NIR', 'SWIR1', 'SWIR2'], + bandsRename: ['ls_blue', 'ls_green', 'ls_red', 'ls_nir', 'ls_swir1', 'ls_swir2' ], + imgProps: ['SENSOR_ID', 'SPACECRAFT_ID'], + imgPropsRename: ['img_id', 'satellite'], + datetimeName: 'date', + datetimeFormat: 'YYYY-MM-dd' }; // Extract zonal statistics per point per image. -var ptsLandsatStats = zonalStats(landsatCol, ptsLandsat, params)   // Filter out observations where image pixels were all masked.   .filter(ee.Filter.notNull(params.bandsRename)); +var ptsLandsatStats = zonalStats(landsatCol, ptsLandsat, params) // Filter out observations where image pixels were all masked. .filter(ee.Filter.notNull(params.bandsRename)); print('Limited Landsat zonal stats table', ptsLandsatStats.limit(50)); +``` The result is a feature collection with a feature for all combinations of plots and images. ###Dealing with Large Collections -If your browser times out, try exporting the results (as described in Chap. F6.2). It’s likely that point feature collections that cover a large area or contain many points (point-image observations) will need to be exported as a batch task by either exporting the final feature collection as an asset or as a CSV/shapefile/GeoJSON to Google Drive or GCS. +If your browser times out, try exporting the results (as described in Chap. F6.2). It’s likely that point feature collections that cover a large area or contain many points (point-image observations) will need to be exported as a batch task by either exporting the final feature collection as an asset or as a CSV/shapefile/GeoJSON to Google Drive or GCS. -Here is how you would export the above Landsat image-point feature collection to an asset and to Google Drive. Run the following code, activate the Code Editor Tasks tab, and then click the Run button. If you don’t specify your own existing folder in Drive, the folder “EEFA_outputs” will be created. +Here is how you would export the above Landsat image-point feature collection to an asset and to Google Drive. Run the following code, activate the Code Editor Tasks tab, and then click the Run button. If you don’t specify your own existing folder in Drive, the folder “EEFA_outputs” will be created. Export.table.toAsset({ -   collection: ptsLandsatStats, -   description: 'EEFA_export_Landsat_to_points', -   assetId: 'EEFA_export_values_to_points' + collection: ptsLandsatStats, + description: 'EEFA_export_Landsat_to_points', + assetId: 'EEFA_export_values_to_points' }); Export.table.toDrive({ -   collection: ptsLandsatStats, -   folder: 'EEFA_outputs', // this will create a new folder if it doesn't exist   description: 'EEFA_export_values_to_points', -   fileFormat: 'CSV' + collection: ptsLandsatStats, + folder: 'EEFA_outputs', // this will create a new folder if it doesn't exist description: 'EEFA_export_values_to_points', + fileFormat: 'CSV' }); -::: {.callout-note} -Code Checkpoint F52b. The book’s repository contains a script that shows what your code should look like at this point. +:::{.callout-note} +Code Checkpoint F52b. The book’s repository contains a script that shows what your code should look like at this point. ::: ## Additional Notes ### Weighted Versus Unweighted Region Reduction -A region used for calculation of zonal statistics often bisects multiple pixels. Should partial pixels be included in zonal statistics? Earth Engine lets you decide by allowing you to define a reducer as either weighted or unweighted (or you can provide per-pixel weight specification as an image band). A weighted reducer will include partial pixels in the zonal statistic calculation by weighting each pixel's contribution according to the fraction of the area intersecting the region. An unweighted reducer, on the other hand, gives equal weight to all pixels whose cell center intersects the region; all other pixels are excluded from calculation of the statistic. +A region used for calculation of zonal statistics often bisects multiple pixels. Should partial pixels be included in zonal statistics? Earth Engine lets you decide by allowing you to define a reducer as either weighted or unweighted (or you can provide per-pixel weight specification as an image band). A weighted reducer will include partial pixels in the zonal statistic calculation by weighting each pixel's contribution according to the fraction of the area intersecting the region. An unweighted reducer, on the other hand, gives equal weight to all pixels whose cell center intersects the region; all other pixels are excluded from calculation of the statistic. -For aggregate reducers like ee.Reducer.mean and ee.Reducer.median, the default mode is weighted, while identifier reducers such as ee.Reducer.min and ee.Reducer.max are unweighted. You can adjust the behavior of weighted reducers by calling unweighted on them, as in ee.Reducer.mean.unweighted. You may also specify the weights by modifying the reducer with splitWeights; however, that is beyond the scope of this book. +For aggregate reducers like ee.Reducer.mean and ee.Reducer.median, the default mode is weighted, while identifier reducers such as ee.Reducer.min and ee.Reducer.max are unweighted. You can adjust the behavior of weighted reducers by calling unweighted on them, as in ee.Reducer.mean.unweighted. You may also specify the weights by modifying the reducer with splitWeights; however, that is beyond the scope of this book. ### Copy Properties to Computed Images Derived, computed images do not retain the properties of their source image, so be sure to copy properties to computed images if you want them included in the region reduction table. For instance, consider the simple computation of unscaling Landsat SR data: +```js // Define a Landsat image. -var img = ee.ImageCollection('LANDSAT/LC08/C02/T1_L2').first(); +var img = ee.ImageCollection('LANDSAT/LC08/C02/T1_L2').first(); // Print its properties. print('All image properties', img.propertyNames()); // Subset the reflectance bands and unscale them. -var computedImg = img.select('SR_B.').multiply(0.0000275).add(-0.2); +var computedImg = img.select('SR_B.').multiply(0.0000275).add(-0.2); // Print the unscaled image's properties. print('Lost original image properties', computedImg.propertyNames()); -Notice how the computed image does not have the source image's properties and only retains the bands information. To fix this, use the copyProperties function to add desired source properties to the derived image. It is best practice to copy only the properties you really need because some properties, such as those containing geometry objects, lists, or feature collections, can significantly increase the computational burden for large collections. +``` +Notice how the computed image does not have the source image's properties and only retains the bands information. To fix this, use the copyProperties function to add desired source properties to the derived image. It is best practice to copy only the properties you really need because some properties, such as those containing geometry objects, lists, or feature collections, can significantly increase the computational burden for large collections. +```js // Subset the reflectance bands and unscale them, keeping selected // source properties. -var computedImg = img.select('SR_B.').multiply(0.0000275).add(-0.2) -   .copyProperties(img, ['system:time_start', 'LANDSAT_PRODUCT_ID']); +var computedImg = img.select('SR_B.').multiply(0.0000275).add(-0.2) + .copyProperties(img, ['system:time_start', 'LANDSAT_PRODUCT_ID']); // Print the unscaled image's properties. print('Selected image properties retained', computedImg .propertyNames()); +``` Now selected properties are included. Use this technique when returning computed, derived images in a mapped function, and in single-image operations. ### Understanding Which Pixels are Included in Polygon Statistics If you want to visualize what pixels are included in a polygon for a region reducer, you can adapt the following code to use your own region (by replacing geometry), dataset, desired scale, and CRS parameters. The important part to note is that the image data you are adding to the map is reprojected using the same scale and CRS as that used in your region reduction (see Fig. F5.2.3). +```js // Define polygon geometry. -var geometry = ee.Geometry.Polygon( -   [ -       [ -           [-118.6019835717645, 37.079867782687884], -           [-118.6019835717645, 37.07838698844939], -           [-118.60036351751951, 37.07838698844939], -           [-118.60036351751951, 37.079867782687884] -       ] -   ], null, false); +var geometry = ee.Geometry.Polygon( + [ + [ + [-118.6019835717645, 37.079867782687884], + [-118.6019835717645, 37.07838698844939], + [-118.60036351751951, 37.07838698844939], + [-118.60036351751951, 37.079867782687884] + ] + ], null, false); // Import the MERIT global elevation dataset. -var elev = ee.Image('MERIT/DEM/v1_0_3'); +var elev = ee.Image('MERIT/DEM/v1_0_3'); // Define desired scale and crs for region reduction (for image display too). -var proj = { -   scale: 90, -   crs: 'EPSG:5070' +var proj = { + scale: 90, + crs: 'EPSG:5070' }; -The count reducer will return how many pixel centers are overlapped by the polygon region, which would be the number of pixels included in any unweighted reducer statistic. You can also visualize which pixels will be included in the reduction by using the toCollection reducer on a latitude/longitude image and adding resulting coordinates as feature geometry. Be sure to specify CRS and scale for both the region reducers and the reprojected layer added to the map (see bullet list below for more details). +``` +The count reducer will return how many pixel centers are overlapped by the polygon region, which would be the number of pixels included in any unweighted reducer statistic. You can also visualize which pixels will be included in the reduction by using the toCollection reducer on a latitude/longitude image and adding resulting coordinates as feature geometry. Be sure to specify CRS and scale for both the region reducers and the reprojected layer added to the map (see bullet list below for more details). +```js // A count reducer will return how many pixel centers are overlapped by the // polygon region. -var count = elev.select(0).reduceRegion({ -   reducer: ee.Reducer.count(), -   geometry: geometry, -   scale: proj.scale,   crs: proj.crs +var count = elev.select(0).reduceRegion({ + reducer: ee.Reducer.count(), + geometry: geometry, + scale: proj.scale, crs: proj.crs }); print('n pixels in the reduction', count.get('dem')); // Make a feature collection of pixel center points for those that are // included in the reduction. -var pixels = ee.Image.pixelLonLat().reduceRegion({ -   reducer: ee.Reducer.toCollection(['lon', 'lat']), -   geometry: geometry, -    scale: proj.scale,   crs: proj.crs +var pixels = ee.Image.pixelLonLat().reduceRegion({ + reducer: ee.Reducer.toCollection(['lon', 'lat']), + geometry: geometry, + scale: proj.scale, crs: proj.crs }); -var pixelsFc = ee.FeatureCollection(pixels.get('features')).map(   function(f) {       return f.setGeometry(ee.Geometry.Point([f.get('lon'), f -           .get('lat') -       ])); -   }); +var pixelsFc = ee.FeatureCollection(pixels.get('features')).map( function(f) { return f.setGeometry(ee.Geometry.Point([f.get('lon'), f + .get('lat') + ])); + }); // Display layers on the map. Map.centerObject(geometry, 18); Map.addLayer( -   elev.reproject({ -       crs: proj.crs, -       scale: proj.scale   }), -   { -       min: 2500, -       max: 3000, -       palette: ['blue', 'white', 'red'] -   }, 'Image'); + elev.reproject({ + crs: proj.crs, + scale: proj.scale }), + { + min: 2500, + max: 3000, + palette: ['blue', 'white', 'red'] + }, 'Image'); Map.addLayer(geometry, { -   color: 'white'}, 'Geometry'); + color: 'white'}, 'Geometry'); Map.addLayer(pixelsFc, { -   color: 'purple'}, 'Pixels in reduction'); + color: 'purple'}, 'Pixels in reduction'); -![](F5/image44.png) +``` +![Fig. F5.2.3 Identifying pixels used in zonal statistics. By mapping the image and vector together, you can see which pixels are included in the unweighted statistic. For this example, three pixels would be included in the statistic because the polygon covers the center point of three pixels.](F5/image44.png) -Fig. F5.2.3 Identifying pixels used in zonal statistics. By mapping the image and vector together, you can see which pixels are included in the unweighted statistic. For this example, three pixels would be included in the statistic because the polygon covers the center point of three pixels. -::: {.callout-note} -Code Checkpoint F52c. The book’s repository contains a script that shows what your code should look like at this point. +:::{.callout-note} +Code Checkpoint F52c. The book’s repository contains a script that shows what your code should look like at this point. ::: Finally, here are some notes on CRS and scale: -* Earth Engine runs reduceRegion using the projection of the image's first band if the CRS is unspecified in the function. For imagery spanning multiple UTM zones, for example, this would lead to different origins. For some functions Earth Engine uses the default EPSG:4326. Therefore, when the opportunity is presented, such as by the reduceRegion function, it is important to specify the scale and CRS explicitly. -* The Map default CRS is EPSG:3857. When looking closely at pixels on the map, the data layer scale and CRS should also be set explicitly. Note that zooming out after setting a relatively small scale when reprojecting may result in memory and/or timeout errors because optimized pyramid layers for each zoom level will not be used. -* Specifying the CRS and scale in both the reduceRegion and addLayer functions allows the map visualization to align with the information printed in the Console. +* Earth Engine runs reduceRegion using the projection of the image's first band if the CRS is unspecified in the function. For imagery spanning multiple UTM zones, for example, this would lead to different origins. For some functions Earth Engine uses the default EPSG:4326. Therefore, when the opportunity is presented, such as by the reduceRegion function, it is important to specify the scale and CRS explicitly. +* The Map default CRS is EPSG:3857. When looking closely at pixels on the map, the data layer scale and CRS should also be set explicitly. Note that zooming out after setting a relatively small scale when reprojecting may result in memory and/or timeout errors because optimized pyramid layers for each zoom level will not be used. +* Specifying the CRS and scale in both the reduceRegion and addLayer functions allows the map visualization to align with the information printed in the Console. * The Earth Engine default, WGS 84 lat long (EPSG:4326), is a generic CRS that works worldwide. The code above reprojects to EPSG:5070, North American Equal Albers, which is a CRS that preserves area for North American locations. Use the CRS that is best for your use case when adapting this to your own project, or maintain (and specify) the CRS of the image using, for example, crs: 'img.projection().crs()'. ## Synthesis {.unnumbered} -Question 1. Look at the MODIS example (Sect. 3.2), which uses the median reducer. Try modifying the reducer to be unweighted, either by specifying unweighted or using an identifier reducer like max. What happens, and why? +Question 1. Look at the MODIS example (Sect. 3.2), which uses the median reducer. Try modifying the reducer to be unweighted, either by specifying unweighted or using an identifier reducer like max. What happens, and why? Question 2. Calculate zonal statistics for your own buffered points or polygons using a raster and reducer of interest. Be sure to consider the spatial scale of the raster and whether a weighted or unweighted reducer would be more appropriate for your interests. If the point or polygon file is stored in a local shapefile or CSV file, first upload the data to your Earth Engine assets. All columns in your vector file, such as the plot name, will be retained through this process. Once you have an Earth Engine table asset ready, import the asset into your script by hovering over the name of the asset and clicking the arrow at the right side, or by calling it in your script with the following code. -var pts = ee.FeatureCollection('users/yourUsername/yourAsset'); +var pts = ee.FeatureCollection('users/yourUsername/yourAsset'); If you prefer to define points or polygons dynamically rather than loading an asset, you can add them to your script using the geometry tools. See Chap. F2.1 and F5.0 for more detail on adding and creating vector data. -Question 3. Try the code from Sect. 4.3 using the MODIS data and the first point from the pts variable. Among other modifications, you will need to create a buffer for the point, take a single MODIS image from the collection, and change visualization parameters. +Question 3. Try the code from Sect. 4.3 using the MODIS data and the first point from the pts variable. Among other modifications, you will need to create a buffer for the point, take a single MODIS image from the collection, and change visualization parameters. * Think about the CRS in the code: The code reprojects to EPSG:5070, but MODIS is collected in the sinusoidal projection SR-ORG:6974. Try that CRS and describe how the image changes. * Is the count reducer weighted or unweighted? Give an example of a circumstance to use a weighted reducer and an example for an unweighted reducer. Specify the buffer size you would use and the spatial resolution of your dataset. -Question 4. In the examples above, only a single ee.Reducer is passed to the zonalStats function, which means that only a single statistic is calculated (for example, zonal mean or median or maximum). What if you want multiple statistics—can you alter the code in Sect. 3.1 to (1) make the point buffer 500 instead of 45; (2) add the reducer parameter to the params dictionary; and (3) as its argument, supply a combined ee.Reducer that will calculate minimum, maximum, standard deviation, and mean statistics? +Question 4. In the examples above, only a single ee.Reducer is passed to the zonalStats function, which means that only a single statistic is calculated (for example, zonal mean or median or maximum). What if you want multiple statistics—can you alter the code in Sect. 3.1 to (1) make the point buffer 500 instead of 45; (2) add the reducer parameter to the params dictionary; and (3) as its argument, supply a combined ee.Reducer that will calculate minimum, maximum, standard deviation, and mean statistics? -To achieve this you’ll need to chain several ee.Reducer.combine functions together. Note that if you accept all the individual ee.Reducer and ee.Reducer.combine function defaults, you’ll run into two problems related to reducer weighting differences, and whether or not the image inputs are shared among the combined set of reducers. How can you manipulate the individual ee.Reducer and ee.Reducer.combine functions to achieve the goal of calculating multiple zonal statistics in one call to the zonalStats function? +To achieve this you’ll need to chain several ee.Reducer.combine functions together. Note that if you accept all the individual ee.Reducer and ee.Reducer.combine function defaults, you’ll run into two problems related to reducer weighting differences, and whether or not the image inputs are shared among the combined set of reducers. How can you manipulate the individual ee.Reducer and ee.Reducer.combine functions to achieve the goal of calculating multiple zonal statistics in one call to the zonalStats function? ## Conclusion {.unnumbered} @@ -1610,19 +1658,19 @@ Miller JD, Thode AE (2007) Quantifying burn severity in a heterogeneous landscap -::: {.callout-tip} +:::{.callout-tip} # Chapter Information ## Author {.unlisted .unnumbered} -  + Ujaval Gandhi ## Overview {.unlisted .unnumbered} -  + This chapter covers advanced techniques for visualizing and analyzing vector data in Earth Engine. There are many ways to visualize feature collections, and you will learn how to pick the appropriate method to create visualizations, such as a choropleth map. We will also cover geoprocessing techniques involving multiple vector layers, such as selecting features in one layer by their proximity to features in another layer and performing spatial joins. @@ -1636,110 +1684,119 @@ This chapter covers advanced techniques for visualizing and analyzing vector dat ## Assumes you know how to:{.unlisted .unnumbered} -* Filter a FeatureCollection to obtain a subset (Chap. F5.0, Chap. F5.1). -* Write a function and map it over a FeatureCollection (Chap. F5.1, Chap. F5.2). +* Filter a FeatureCollection to obtain a subset (Chap. F5.0, Chap. F5.1). +* Write a function and map it over a FeatureCollection (Chap. F5.1, Chap. F5.2). ## Visualizing Feature Collections -There is a distinct difference between how rasters and vectors are visualized. While images are typically visualized based on pixel values, vector layers use feature properties (i.e., attributes) to create a visualization. Vector layers are rendered on the Map by assigning a value to the red, green, and blue channels for each pixel on the screen based on the geometry and attributes of the features. The functions used for vector data visualization in Earth Engine are listed below in increasing order of complexity. +There is a distinct difference between how rasters and vectors are visualized. While images are typically visualized based on pixel values, vector layers use feature properties (i.e., attributes) to create a visualization. Vector layers are rendered on the Map by assigning a value to the red, green, and blue channels for each pixel on the screen based on the geometry and attributes of the features. The functions used for vector data visualization in Earth Engine are listed below in increasing order of complexity. -* Map.addLayer: As with raster layers, you can add a FeatureCollection to the Map by specifying visualization parameters. This method supports only one visualization parameter: color. All features are rendered with the specified color. -* draw: This function supports the parameters pointRadius and strokeWidth in addition to color. It renders all features of the layer with the specified parameters. -* paint: This is a more powerful function that can render each feature with a different color and width based on the values in the specified property. +* Map.addLayer: As with raster layers, you can add a FeatureCollection to the Map by specifying visualization parameters. This method supports only one visualization parameter: color. All features are rendered with the specified color. +* draw: This function supports the parameters pointRadius and strokeWidth in addition to color. It renders all features of the layer with the specified parameters. +* paint: This is a more powerful function that can render each feature with a different color and width based on the values in the specified property. * style: This is the most versatile function. It can apply a different style to each feature, including color, pointSize, pointShape, width, fillColor, and lineType. In the exercises below, we will learn how to use each of these functions and see how they can generate different types of maps. ### Creating a Choropleth Map -We will use the TIGER: US Census Blocks layer, which stores census block boundaries and their characteristics within the United States, along with the San Francisco neighborhoods layer from Chap. F5.0 to create a population density map for the city of San Francisco. +We will use the TIGER: US Census Blocks layer, which stores census block boundaries and their characteristics within the United States, along with the San Francisco neighborhoods layer from Chap. F5.0 to create a population density map for the city of San Francisco. -We start by loading the census blocks and San Francisco neighborhoods layers. We use ee.Filter.bounds to filter the census blocks layer to the San Francisco boundary. +We start by loading the census blocks and San Francisco neighborhoods layers. We use ee.Filter.bounds to filter the census blocks layer to the San Francisco boundary. -var blocks = ee.FeatureCollection('TIGER/2010/Blocks'); -var roads = ee.FeatureCollection('TIGER/2016/Roads'); -var sfNeighborhoods = ee.FeatureCollection(   'projects/gee-book/assets/F5-0/SFneighborhoods'); +var blocks = ee.FeatureCollection('TIGER/2010/Blocks'); +var roads = ee.FeatureCollection('TIGER/2016/Roads'); +var sfNeighborhoods = ee.FeatureCollection( 'projects/gee-book/assets/F5-0/SFneighborhoods'); -var geometry = sfNeighborhoods.geometry(); +var geometry = sfNeighborhoods.geometry(); Map.centerObject(geometry); +```js // Filter blocks to the San Francisco boundary. -var sfBlocks = blocks.filter(ee.Filter.bounds(geometry)); +var sfBlocks = blocks.filter(ee.Filter.bounds(geometry)); -The simplest way to visualize this layer is to use Map.addLayer (Fig. F5.3.1). We can specify a color value in the visParams parameter of the function. Each census block polygon will be rendered with stroke and fill of the specified color. The fill color is the same as the stroke color but has a 66% opacity. +``` +The simplest way to visualize this layer is to use Map.addLayer (Fig. F5.3.1). We can specify a color value in the visParams parameter of the function. Each census block polygon will be rendered with stroke and fill of the specified color. The fill color is the same as the stroke color but has a 66% opacity. +```js // Visualize with a single color. Map.addLayer(sfBlocks, { -   color: '#de2d26'}, 'Census Blocks (single color)'); + color: '#de2d26'}, 'Census Blocks (single color)'); -![](F5/image34.png) +``` +![Fig. F5.3.1 San Francisco census blocks](F5/image34.png) -Fig. F5.3.1 San Francisco census blocks -The census blocks table has a property named 'pop10' containing the population totals as of the 2010 census. We can use this to create a choropleth map showing population density. We first need to compute the population density for each feature and add it as a property. To add a new property to each feature, we can map a function over the FeatureCollection and calculate the new property called 'pop_density'. Earth Engine provides the area function, which can calculate the area of a feature in square meters. We convert it to square miles and calculate the population density per square mile. +The census blocks table has a property named 'pop10' containing the population totals as of the 2010 census. We can use this to create a choropleth map showing population density. We first need to compute the population density for each feature and add it as a property. To add a new property to each feature, we can map a function over the FeatureCollection and calculate the new property called 'pop_density'. Earth Engine provides the area function, which can calculate the area of a feature in square meters. We convert it to square miles and calculate the population density per square mile. +```js // Add a pop_density column. -var sfBlocks = sfBlocks.map(function(f) {   // Get the polygon area in square miles.   var area_sqmi = f.area().divide(2.59e6);   var population = f.get('pop10');   // Calculate population density.   var density = ee.Number(population).divide(area_sqmi);   return f.set({       'area_sqmi': area_sqmi,       'pop_density': density -   }); +var sfBlocks = sfBlocks.map(function(f) { // Get the polygon area in square miles. var area_sqmi = f.area().divide(2.59e6); var population = f.get('pop10'); // Calculate population density. var density = ee.Number(population).divide(area_sqmi); return f.set({ 'area_sqmi': area_sqmi, 'pop_density': density + }); }); -Now we can use the paint function to create an image from this FeatureCollection using the pop_density property. The paint function needs an empty image that needs to be cast to the appropriate data type. Let’s use the aggregate_stats function to calculate basic statistics for the given column of a FeatureCollection. +``` +Now we can use the paint function to create an image from this FeatureCollection using the pop_density property. The paint function needs an empty image that needs to be cast to the appropriate data type. Let’s use the aggregate_stats function to calculate basic statistics for the given column of a FeatureCollection. +```js // Calculate the statistics of the newly computed column. -var stats = sfBlocks.aggregate_stats('pop_density'); +var stats = sfBlocks.aggregate_stats('pop_density'); print(stats); +``` You will see that the population density values have a large range. We also have values that are greater than 100,000, so we need to make sure we select a data type that can store values of this size. We create an empty image and cast it to int32, which is able to hold large integer values. D -The result is an image with pixel values representing the population density of the polygons. We can now use the standard image visualization method to add this layer to the Map (Fig. F5.3.2). Then, we need to determine minimum and maximum values for the visualization parameters.A reliable technique to produce a good visualization is to find minimum and maximum values that are within one standard deviation. From the statistics that we calculated earlier, we can estimate good minimum and maximum values to be 0 and 50000, respectively. +The result is an image with pixel values representing the population density of the polygons. We can now use the standard image visualization method to add this layer to the Map (Fig. F5.3.2). Then, we need to determine minimum and maximum values for the visualization parameters.A reliable technique to produce a good visualization is to find minimum and maximum values that are within one standard deviation. From the statistics that we calculated earlier, we can estimate good minimum and maximum values to be 0 and 50000, respectively. -var palette = ['fee5d9', 'fcae91', 'fb6a4a', 'de2d26', 'a50f15']; -var visParams = { -   min: 0, -   max: 50000, -   palette: palette +var palette = ['fee5d9', 'fcae91', 'fb6a4a', 'de2d26', 'a50f15']; +var visParams = { + min: 0, + max: 50000, + palette: palette }; -Map.addLayer(sfBlocksPaint.clip(geometry), visParams,   'Population Density'); +Map.addLayer(sfBlocksPaint.clip(geometry), visParams, 'Population Density'); -![](F5/image41.png) +![Fig. F5.3.2 San Francisco population density](F5/image41.png) -Fig. F5.3.2 San Francisco population density ### Creating a Categorical Map -Continuing the exploration of styling methods, we will now learn about draw and style. These are the preferred methods of styling for points and line layers. Let’s see how we can visualize the TIGER: US Census Roads layer to create a categorical map. +Continuing the exploration of styling methods, we will now learn about draw and style. These are the preferred methods of styling for points and line layers. Let’s see how we can visualize the TIGER: US Census Roads layer to create a categorical map. -We start by filtering the roads layer to the San Francisco boundary and using Map.addLayer to visualize it. +We start by filtering the roads layer to the San Francisco boundary and using Map.addLayer to visualize it. +```js // Filter roads to San Francisco boundary. -var sfRoads = roads.filter(ee.Filter.bounds(geometry)); +var sfRoads = roads.filter(ee.Filter.bounds(geometry)); Map.addLayer(sfRoads, { -   color: 'blue'}, 'Roads (default)'); + color: 'blue'}, 'Roads (default)'); -The default visualization renders each line using a width of 2 pixels. The draw function provides a way to specify a different line width. Let’s use it to render the layer with the same color as before but with a line width of 1 pixel (Fig. F5.3.3). +``` +The default visualization renders each line using a width of 2 pixels. The draw function provides a way to specify a different line width. Let’s use it to render the layer with the same color as before but with a line width of 1 pixel (Fig. F5.3.3). +```js // Visualize with draw(). -var sfRoadsDraw = sfRoads.draw({ -   color: 'blue', -   strokeWidth: 1 +var sfRoadsDraw = sfRoads.draw({ + color: 'blue', + strokeWidth: 1 }); Map.addLayer(sfRoadsDraw, {}, 'Roads (Draw)'); +``` ![](F5/image28.png) -![](F5/image31.png) +![Fig. F5.3.3 San Francisco roads rendered with a line width of 2 pixels (left) and and a line width of 1 pixel (right)](F5/image31.png) -Fig. F5.3.3 San Francisco roads rendered with a line width of 2 pixels (left) and and a line width of 1 pixel (right) -The road layer has a column called “MTFCC” (standing for the MAF/TIGER Feature Class Code). This contains the road priority codes, representing the various types of roads, such as primary and secondary. We can use this information to render each road segment according to its priority. The draw function doesn’t allow us to specify different styles for each feature. Instead, we need to make use of the style function. +The road layer has a column called “MTFCC” (standing for the MAF/TIGER Feature Class Code). This contains the road priority codes, representing the various types of roads, such as primary and secondary. We can use this information to render each road segment according to its priority. The draw function doesn’t allow us to specify different styles for each feature. Instead, we need to make use of the style function. -The column contains string values indicating different road types as indicated in Table F5.3.1. This full list is available at the MAF/TIGER Feature Class Code Definitions page on the US Census Bureau website. +The column contains string values indicating different road types as indicated in Table F5.3.1. This full list is available at the MAF/TIGER Feature Class Code Definitions page on the US Census Bureau website. -Table F5.3.1 Census Bureau road priority codes +Table F5.3.1 Census Bureau road priority codes MTFCC @@ -1807,7 +1864,7 @@ Road Median Let’s say we want to create a map with rules based on the MTFCC values shown in Table F5.3.2. -Table F5.3.2 Styling Parameters for Road Priority Codes +Table F5.3.2 Styling Parameters for Road Priority Codes MTFCC @@ -1841,29 +1898,28 @@ Gray Let’s define a dictionary containing the styling information. -var styles = ee.Dictionary({   'S1100': {       'color': 'blue',       'width': 3   },   'S1200': {       'color': 'green',       'width': 2   },   'S1400': {       'color': 'orange',       'width': 1   } -});var defaultStyle = { -   color: 'gray',   'width': 1 +var styles = ee.Dictionary({ 'S1100': { 'color': 'blue', 'width': 3 }, 'S1200': { 'color': 'green', 'width': 2 }, 'S1400': { 'color': 'orange', 'width': 1 } +});var defaultStyle = { + color: 'gray', 'width': 1 }; -The style function needs a property in the FeatureCollection that contains a dictionary with the style parameters. This allows you to specify a different style for each feature. To create a new property, we map a function over the FeatureCollection and assign an appropriate style dictionary to a new property named 'style'. Note the use of the get function, which allows us to fetch the value for a key in the dictionary. It also takes a default value in case the specified key does not exist. We make use of this to assign different styles to the three road classes specified in Table 5.3.2 and a default style to all others. +The style function needs a property in the FeatureCollection that contains a dictionary with the style parameters. This allows you to specify a different style for each feature. To create a new property, we map a function over the FeatureCollection and assign an appropriate style dictionary to a new property named 'style'. Note the use of the get function, which allows us to fetch the value for a key in the dictionary. It also takes a default value in case the specified key does not exist. We make use of this to assign different styles to the three road classes specified in Table 5.3.2 and a default style to all others. -var sfRoads = sfRoads.map(function(f) {   var classcode = f.get('mtfcc');   var style = styles.get(classcode, defaultStyle);   return f.set('style', style); +var sfRoads = sfRoads.map(function(f) { var classcode = f.get('mtfcc'); var style = styles.get(classcode, defaultStyle); return f.set('style', style); }); -Our collection is now ready to be styled. We call the style function to specify the property that contains the dictionary of style parameters. The output of the style function is an RGB image rendered from the FeatureCollection (Fig. F5.3.4). +Our collection is now ready to be styled. We call the style function to specify the property that contains the dictionary of style parameters. The output of the style function is an RGB image rendered from the FeatureCollection (Fig. F5.3.4). -var sfRoadsStyle = sfRoads.style({ -   styleProperty: 'style' +var sfRoadsStyle = sfRoads.style({ + styleProperty: 'style' }); Map.addLayer(sfRoadsStyle.clip(geometry), {}, 'Roads (Style)'); -![](F5/image46.png) +![Fig. F5.3.4 San Francisco roads rendered according to road priority](F5/image46.png) -Fig. F5.3.4 San Francisco roads rendered according to road priority -::: {.callout-note} -Code Checkpoint F53a. The book’s repository contains a script that shows what your code should look like at this point. +:::{.callout-note} +Code Checkpoint F53a. The book’s repository contains a script that shows what your code should look like at this point. ::: Save your script for your own future use, as outlined in Chap. F1.0. Then, refresh the Code Editor to begin with a new script for the next section. @@ -1871,10 +1927,10 @@ Save your script for your own future use, as outlined in Chap. F1.0. Then, refre Earth Engine was designed as a platform for processing raster data, and that is where it shines. Over the years, it has acquired advanced vector data processing capabilities, and users are now able to carry out complex geoprocessing tasks within Earth Engine. You can leverage the distributed processing power of Earth Engine to process large vector layers in parallel. -This section shows how you can do spatial queries and spatial joins using multiple large feature collections. This requires the use of joins. As described for Image Collections in Chap. F4.9, a join allows you to match every item in a collection with items in another collection based on certain conditions. While you can achieve similar results using map and filter, joins perform better and give you more flexibility. We need to define the following items to perform a join on two collections. +This section shows how you can do spatial queries and spatial joins using multiple large feature collections. This requires the use of joins. As described for Image Collections in Chap. F4.9, a join allows you to match every item in a collection with items in another collection based on certain conditions. While you can achieve similar results using map and filter, joins perform better and give you more flexibility. We need to define the following items to perform a join on two collections. -1. Filter: A filter defines the condition used to select the features from the two collections. There is a suite of filters in the ee.Filters module that work on two collections, such as ee.Filter.equals and ee.Filter.withinDistance. -2. Join type: While the filter determines which features will be joined, the join type determines how they will be joined. There are many join types, including simple join, inner join, and save-all join. +1. Filter: A filter defines the condition used to select the features from the two collections. There is a suite of filters in the ee.Filters module that work on two collections, such as ee.Filter.equals and ee.Filter.withinDistance. +2. Join type: While the filter determines which features will be joined, the join type determines how they will be joined. There are many join types, including simple join, inner join, and save-all join. Joins are one of the harder skills to master, but doing so will help you perform many complex analysis tasks within Earth Engine. We will go through practical examples that will help you understand these concepts and the workflow better. @@ -1884,162 +1940,164 @@ In this section, we will learn how to select features from one layer that are wi We start by loading the census blocks and roads collections and filtering the roads layer to the San Francisco boundary. -var blocks = ee.FeatureCollection('TIGER/2010/Blocks'); -var roads = ee.FeatureCollection('TIGER/2016/Roads'); -var sfNeighborhoods = ee.FeatureCollection(   'projects/gee-book/assets/F5-0/SFneighborhoods'); +var blocks = ee.FeatureCollection('TIGER/2010/Blocks'); +var roads = ee.FeatureCollection('TIGER/2016/Roads'); +var sfNeighborhoods = ee.FeatureCollection( 'projects/gee-book/assets/F5-0/SFneighborhoods'); -var geometry = sfNeighborhoods.geometry(); +var geometry = sfNeighborhoods.geometry(); Map.centerObject(geometry); +```js // Filter blocks and roads to San Francisco boundary. -var sfBlocks = blocks.filter(ee.Filter.bounds(geometry)); -var sfRoads = roads.filter(ee.Filter.bounds(geometry)); +var sfBlocks = blocks.filter(ee.Filter.bounds(geometry)); +var sfRoads = roads.filter(ee.Filter.bounds(geometry)); -As we want to select all blocks within 1 km of an interstate highway, we first filter the sfRoads collection to select all segments with the rttyp property value of I. +``` +As we want to select all blocks within 1 km of an interstate highway, we first filter the sfRoads collection to select all segments with the rttyp property value of I. -var interstateRoads = sfRoads.filter(ee.Filter.eq('rttyp', 'I')); +var interstateRoads = sfRoads.filter(ee.Filter.eq('rttyp', 'I')); -We use the draw function to visualize the sfBlocks and interstateRoads layers (Fig. F5.3.5). +We use the draw function to visualize the sfBlocks and interstateRoads layers (Fig. F5.3.5). -var sfBlocksDrawn = sfBlocks.draw({ -       color: 'gray', -       strokeWidth: 1   }) -   .clip(geometry); +var sfBlocksDrawn = sfBlocks.draw({ + color: 'gray', + strokeWidth: 1 }) + .clip(geometry); Map.addLayer(sfBlocksDrawn, {}, 'All Blocks'); -var interstateRoadsDrawn = interstateRoads.draw({ -       color: 'blue', -       strokeWidth: 3   }) -   .clip(geometry); +var interstateRoadsDrawn = interstateRoads.draw({ + color: 'blue', + strokeWidth: 3 }) + .clip(geometry); Map.addLayer(interstateRoadsDrawn, {}, 'Interstate Roads'); -![](F5/image2.png) +![Fig. F5.3.5 San Francisco blocks and interstate highways](F5/image2.png) -Fig. F5.3.5 San Francisco blocks and interstate highways -Let’s define a join that will select all the features from the sfBlocks layer that are within 1 km of any feature from the interstateRoads layer. We start by defining a filter using the ee.Filter.withinDistance filter. We want to compare the geometries of features in both layers, so we use a special property called '.geo' to compare the collections. By default, the filter will work with exact distances between the geometries. If your analysis does not require a very precise tolerance of spatial uncertainty, specifying a small non-zero maxError distance value will help speed up the spatial operations. A larger tolerance also helps when testing or debugging code so you can get the result quickly instead of waiting longer for a more precise output. +Let’s define a join that will select all the features from the sfBlocks layer that are within 1 km of any feature from the interstateRoads layer. We start by defining a filter using the ee.Filter.withinDistance filter. We want to compare the geometries of features in both layers, so we use a special property called '.geo' to compare the collections. By default, the filter will work with exact distances between the geometries. If your analysis does not require a very precise tolerance of spatial uncertainty, specifying a small non-zero maxError distance value will help speed up the spatial operations. A larger tolerance also helps when testing or debugging code so you can get the result quickly instead of waiting longer for a more precise output. -var joinFilter = ee.Filter.withinDistance({ -   distance: 1000, -   leftField: '.geo', -   rightField: '.geo', -   maxError: 10 +var joinFilter = ee.Filter.withinDistance({ + distance: 1000, + leftField: '.geo', + rightField: '.geo', + maxError: 10 }); -We will use a simple join as we just want features from the first (primary) collection that match the features from the other (secondary) collection. +We will use a simple join as we just want features from the first (primary) collection that match the features from the other (secondary) collection. -var closeBlocks = ee.Join.simple().apply({ -   primary: sfBlocks, -   secondary: interstateRoads, -   condition: joinFilter +var closeBlocks = ee.Join.simple().apply({ + primary: sfBlocks, + secondary: interstateRoads, + condition: joinFilter }); We can visualize the results in a different color and verify that the join worked as expected (Fig. F5.3.6). -var closeBlocksDrawn = closeBlocks.draw({ -       color: 'orange', -       strokeWidth: 1   }) -   .clip(geometry); +var closeBlocksDrawn = closeBlocks.draw({ + color: 'orange', + strokeWidth: 1 }) + .clip(geometry); Map.addLayer(closeBlocksDrawn, {}, 'Blocks within 1km'); -![](F5/image40.png) +![Fig. F5.3.6 Selected blocks within 1 km of an interstate highway](F5/image40.png) -Fig. F5.3.6 Selected blocks within 1 km of an interstate highway ### Spatial Joins -A spatial join allows you to query two collections based on the spatial relationship. We will now implement a spatial join to count points in polygons. We will work with a dataset of tree locations in San Francisco and polygons of neighborhoods to produce a CSV file with the total number of trees in each neighborhood. +A spatial join allows you to query two collections based on the spatial relationship. We will now implement a spatial join to count points in polygons. We will work with a dataset of tree locations in San Francisco and polygons of neighborhoods to produce a CSV file with the total number of trees in each neighborhood. -The San Francisco Open Data Portal maintains a street tree map dataset that has a list of street trees with their latitude and longitude. We will also use the San Francisco neighborhood dataset from the same portal. We downloaded, processed, and uploaded these layers as Earth Engine assets for use in this exercise. We start by loading both layers and using the paint and style functions, covered in Sect. 1, to visualize them (Fig. F5.3.7). +The San Francisco Open Data Portal maintains a street tree map dataset that has a list of street trees with their latitude and longitude. We will also use the San Francisco neighborhood dataset from the same portal. We downloaded, processed, and uploaded these layers as Earth Engine assets for use in this exercise. We start by loading both layers and using the paint and style functions, covered in Sect. 1, to visualize them (Fig. F5.3.7). -var sfNeighborhoods = ee.FeatureCollection(   'projects/gee-book/assets/F5-0/SFneighborhoods'); -var sfTrees = ee.FeatureCollection(   'projects/gee-book/assets/F5-3/SFTrees'); +var sfNeighborhoods = ee.FeatureCollection( 'projects/gee-book/assets/F5-0/SFneighborhoods'); +var sfTrees = ee.FeatureCollection( 'projects/gee-book/assets/F5-3/SFTrees'); +```js // Use paint() to visualize the polygons with only outline -var sfNeighborhoodsOutline = ee.Image().byte().paint({ -   featureCollection: sfNeighborhoods, -   color: 1, -   width: 3 +var sfNeighborhoodsOutline = ee.Image().byte().paint({ + featureCollection: sfNeighborhoods, + color: 1, + width: 3 }); Map.addLayer(sfNeighborhoodsOutline, { -       palette: ['blue'] -   },   'SF Neighborhoods'); + palette: ['blue'] + }, 'SF Neighborhoods'); // Use style() to visualize the points -var sfTreesStyled = sfTrees.style({ -   color: 'green', -   pointSize: 2, -   pointShape: 'triangle', -   width: 2 +var sfTreesStyled = sfTrees.style({ + color: 'green', + pointSize: 2, + pointShape: 'triangle', + width: 2 }); Map.addLayer(sfTreesStyled, {}, 'SF Trees'); -![](F5/image35.png) +``` +![Fig. F5.3.7 San Francisco neighborhoods and trees](F5/image35.png) -Fig. F5.3.7 San Francisco neighborhoods and trees -To find the tree points in each neighborhood polygon, we will use an ee.Filter.intersects filter. +To find the tree points in each neighborhood polygon, we will use an ee.Filter.intersects filter. -var intersectFilter = ee.Filter.intersects({ -   leftField: '.geo', -   rightField: '.geo', -   maxError: 10 +var intersectFilter = ee.Filter.intersects({ + leftField: '.geo', + rightField: '.geo', + maxError: 10 }); -We need a join that can give us a list of all tree features that intersect each neighborhood polygon, so we need to use a saving join. A saving join will find all the features from the secondary collection that match the filter and store them in a property in the primary collection. Once you apply this join, you will get a version of the primary collection with an additional property that has the matching features from the secondary collection. Here we use the ee.Join.saveAll join, since we want to store all matching features. We specify the matchesKey property that will be added to each feature with the results. +We need a join that can give us a list of all tree features that intersect each neighborhood polygon, so we need to use a saving join. A saving join will find all the features from the secondary collection that match the filter and store them in a property in the primary collection. Once you apply this join, you will get a version of the primary collection with an additional property that has the matching features from the secondary collection. Here we use the ee.Join.saveAll join, since we want to store all matching features. We specify the matchesKey property that will be added to each feature with the results. -var saveAllJoin = ee.Join.saveAll({ -   matchesKey: 'trees', +var saveAllJoin = ee.Join.saveAll({ + matchesKey: 'trees', }); Let’s apply the join and print the first feature of the resulting collection to verify (Fig. F5.3.8). -var joined = saveAllJoin -   .apply(sfNeighborhoods, sfTrees, intersectFilter); +var joined = saveAllJoin + .apply(sfNeighborhoods, sfTrees, intersectFilter); print(joined.first()); -![](F5/image1.png) +![Fig. F5.3.8 Result of the save-all join](F5/image1.png) -Fig. F5.3.8 Result of the save-all join -You will see that each feature of the sfNeighborhoods collection now has an additional property called trees. This contains all the features from the sfTrees collection that were matched using the intersectFilter. We can now map a function over the results and post-process the collection. As our analysis requires the computation of the total number of trees in each neighborhood, we extract the matching features and use the size function to get the count (Fig. F5.3.9). +You will see that each feature of the sfNeighborhoods collection now has an additional property called trees. This contains all the features from the sfTrees collection that were matched using the intersectFilter. We can now map a function over the results and post-process the collection. As our analysis requires the computation of the total number of trees in each neighborhood, we extract the matching features and use the size function to get the count (Fig. F5.3.9). +```js // Calculate total number of trees within each feature. -var sfNeighborhoods = joined.map(function(f) {   var treesWithin = ee.List(f.get('trees'));   var totalTrees = ee.FeatureCollection(treesWithin).size();   return f.set('total_trees', totalTrees); +var sfNeighborhoods = joined.map(function(f) { var treesWithin = ee.List(f.get('trees')); var totalTrees = ee.FeatureCollection(treesWithin).size(); return f.set('total_trees', totalTrees); }); print(sfNeighborhoods.first()); -![](F5/image18.png) +``` +![Fig. F5.3.9 Final FeatureCollection with the new property](F5/image18.png) -Fig. F5.3.9 Final FeatureCollection with the new property -The results now have a property called total_trees containing the count of intersecting trees in each neighborhood polygon. +The results now have a property called total_trees containing the count of intersecting trees in each neighborhood polygon. -The final step in the analysis is to export the results as a CSV file using the Export.table.toDrive function. Note that as described in detail in F6.2, you should output only the columns you need to the CSV file. Suppose we do not need all the properties to appear in the output; imagine that wedo not need the trees property, for example, in the output. In that case, we can create only those columns we want in the manner below, by specifying the other selectors parameters with the list of properties to export. +The final step in the analysis is to export the results as a CSV file using the Export.table.toDrive function. Note that as described in detail in F6.2, you should output only the columns you need to the CSV file. Suppose we do not need all the properties to appear in the output; imagine that wedo not need the trees property, for example, in the output. In that case, we can create only those columns we want in the manner below, by specifying the other selectors parameters with the list of properties to export. +```js // Export the results as a CSV. Export.table.toDrive({ -   collection: sfNeighborhoods, -   description: 'SF_Neighborhood_Tree_Count', -   folder: 'earthengine', -   fileNamePrefix: 'tree_count', -   fileFormat: 'CSV', -   selectors: ['nhood', 'total_trees'] + collection: sfNeighborhoods, + description: 'SF_Neighborhood_Tree_Count', + folder: 'earthengine', + fileNamePrefix: 'tree_count', + fileFormat: 'CSV', + selectors: ['nhood', 'total_trees'] }); +``` The final result is a CSV file with the neighborhood names and total numbers of trees counted using the join (Fig. F5.3.10). -![](F5/image3.png) +![Fig. F5.3.10 Exported CSV file with tree counts for San Francisco neighborhoods](F5/image3.png) -Fig. F5.3.10 Exported CSV file with tree counts for San Francisco neighborhoods -::: {.callout-note} -Code Checkpoint F53b. The book’s repository contains a script that shows what your code should look like at this point. +:::{.callout-note} +Code Checkpoint F53b. The book’s repository contains a script that shows what your code should look like at this point. ::: ## Synthesis {.unnumbered} -Assignment 1. What join would you use if you wanted to know which neighborhood each tree belongs to? Modify the code above to do a join and post-process the result to add a neighborhood property to each tree point. Export the results as a shapefile. +Assignment 1. What join would you use if you wanted to know which neighborhood each tree belongs to? Modify the code above to do a join and post-process the result to add a neighborhood property to each tree point. Export the results as a shapefile. ## Conclusion {.unnumbered} -This chapter covered visualization and analysis using vector data in Earth Engine. You should now understand different functions for FeatureCollection visualization and be able to create thematic maps with vector layers. You also learned techniques for doing spatial queries and spatial joins within Earth Engine. Earth Engine is capable of handling large feature collections and can be effectively used for many spatial analysis tasks. \ No newline at end of file +This chapter covered visualization and analysis using vector data in Earth Engine. You should now understand different functions for FeatureCollection visualization and be able to create thematic maps with vector layers. You also learned techniques for doing spatial queries and spatial joins within Earth Engine. Earth Engine is capable of handling large feature collections and can be effectively used for many spatial analysis tasks. \ No newline at end of file diff --git a/F6.qmd b/F6.qmd index 285d457..a9a91ee 100644 --- a/F6.qmd +++ b/F6.qmd @@ -15,7 +15,7 @@ Although you now know the most basic fundamentals of Earth Engine, there is stil -::: {.callout-tip} +:::{.callout-tip} # Chapter Information ## Author {.unlisted .unnumbered} @@ -44,10 +44,10 @@ This chapter should help users of Earth Engine to better understand raster data * Import images and image collections, filter, and visualize (Part F1). -* Write a function and map it over an ImageCollection (Chap. F4.0). -* Inspect an Image and an ImageCollection, as well as their properties (Chap. F4.1). +* Write a function and map it over an ImageCollection (Chap. F4.0). +* Inspect an Image and an ImageCollection, as well as their properties (Chap. F4.1). -::: +::: ## Introduction {.unlisted .unnumbered} @@ -72,194 +72,202 @@ In this section we will explore examples of colormaps to visualize raster data. There are multiple types of colormaps, each used for a different purpose. These include the following: -Sequential: These are probably the most commonly used colormaps, and are useful for ordinal, interval, and ratio data. Also referred to as a linear colormap, a sequential colormap looks like the viridis colormap (Fig. F6.0.1) from matplotlib. It is popular because it is a perceptual uniform colormap, where an equal interval in values is mapped to an equal interval in the perceptual colorspace. If you have a ratio variable where zero means nothing, you can use a sequential colormap starting at white, transparent, or, when you have a black background, at black—for example, the turku colormap from Crameri (Fig. F6.0.1). You can use this for variables like population count or gross domestic product. +Sequential: These are probably the most commonly used colormaps, and are useful for ordinal, interval, and ratio data. Also referred to as a linear colormap, a sequential colormap looks like the viridis colormap (Fig. F6.0.1) from matplotlib. It is popular because it is a perceptual uniform colormap, where an equal interval in values is mapped to an equal interval in the perceptual colorspace. If you have a ratio variable where zero means nothing, you can use a sequential colormap starting at white, transparent, or, when you have a black background, at black—for example, the turku colormap from Crameri (Fig. F6.0.1). You can use this for variables like population count or gross domestic product. -Diverging: This type of colormap is used for visualizing data where you have positive and negative values and where zero has a meaning. Later in this tutorial, we will use the balance colormap from the cmocean package (Fig. F6.0.1) to show temperature change. +Diverging: This type of colormap is used for visualizing data where you have positive and negative values and where zero has a meaning. Later in this tutorial, we will use the balance colormap from the cmocean package (Fig. F6.0.1) to show temperature change. -Circular: Some variables are periodic, returning to the same value after a period of time. For example, the season, angle, and time of day are typically represented as circular variables. For variables like this, a circular colormap is designed to represent the first and last values with the same color. An example is the circular cet-c2 colormap (Fig. F6.0.1) from the colorcet package. +Circular: Some variables are periodic, returning to the same value after a period of time. For example, the season, angle, and time of day are typically represented as circular variables. For variables like this, a circular colormap is designed to represent the first and last values with the same color. An example is the circular cet-c2 colormap (Fig. F6.0.1) from the colorcet package. -Semantic: Some colormaps do not map to arbitrary colors but choose colors that provide meaning. We refer to these as semantic colormaps. Later in this tutorial, we will use the ice colormap (Fig. F6.0.1) from the cmocean package for our ice example. +Semantic: Some colormaps do not map to arbitrary colors but choose colors that provide meaning. We refer to these as semantic colormaps. Later in this tutorial, we will use the ice colormap (Fig. F6.0.1) from the cmocean package for our ice example. -![](F6/image40.png) +![Fig. F6.0.1 Examples of colormaps from a variety of packages: viridis from matplotlib, turku from Crameri, balance from cmocean, cet-c2 from colorcet and ice from cmocean](F6/image40.png) -Fig. F6.0.1 Examples of colormaps from a variety of packages: viridis from matplotlib, turku from Crameri, balance from cmocean, cet-c2 from colorcet and ice from cmocean Popular sources of colormaps include: -* cmocean (semantic perceptual uniform colormaps for geophysical applications) +* cmocean (semantic perceptual uniform colormaps for geophysical applications) * colorcet (set of perceptual colormaps with varying colors and saturation) * cpt-city (comprehensive overview of colormaps, * colorbrewer (colormaps with variety of colors) * Crameri (stylish colormaps for dark and light themes) -Our first example in this section applies a diverging colormap to temperature. +Our first example in this section applies a diverging colormap to temperature. +```js // Load the ERA5 reanalysis monthly means. -var era5 = ee.ImageCollection('ECMWF/ERA5_LAND/MONTHLY'); +var era5 = ee.ImageCollection('ECMWF/ERA5_LAND/MONTHLY'); // Load the palettes package. -var palettes = require('users/gena/packages:palettes'); +var palettes = require('users/gena/packages:palettes'); // Select temperature near ground. era5 = era5.select('temperature_2m'); -Now we can visualize the data. Here we have a temperature difference. That means that zero has a special meaning. By using a divergent colormap we can give zero the color white, which denotes that there is no significant difference. Here we will use the colormap Balance from the cmocean package. The color red is associated with warmth, and the color blue is associated with cold. We will choose the minimum and maximum values for the palette to be symmetric around zero (-2, 2) so that white appears in the correct place.  For comparison we also visualize the data with a simple ['blue', 'white', 'red'] palette. As you can see (Fig. F6.0.2), the Balance colormap has a more elegant and professional feel to it, because it uses a perceptual uniform palette and both saturation and value. +``` +Now we can visualize the data. Here we have a temperature difference. That means that zero has a special meaning. By using a divergent colormap we can give zero the color white, which denotes that there is no significant difference. Here we will use the colormap Balance from the cmocean package. The color red is associated with warmth, and the color blue is associated with cold. We will choose the minimum and maximum values for the palette to be symmetric around zero (-2, 2) so that white appears in the correct place. For comparison we also visualize the data with a simple ['blue', 'white', 'red'] palette. As you can see (Fig. F6.0.2), the Balance colormap has a more elegant and professional feel to it, because it uses a perceptual uniform palette and both saturation and value. +```js // Choose a diverging colormap for anomalies. -var balancePalette = palettes.cmocean.Balance[7]; -var threeColorPalette = ['blue', 'white', 'red']; +var balancePalette = palettes.cmocean.Balance[7]; +var threeColorPalette = ['blue', 'white', 'red']; // Show the palette in the Inspector window. palettes.showPalette('temperature anomaly', balancePalette); palettes.showPalette('temperature anomaly', threeColorPalette); // Select 2 time windows of 10 years. -var era5_1980 = era5.filterDate('1981-01-01', '1991-01-01').mean(); -var era5_2010 = era5.filterDate('2011-01-01', '2020-01-01').mean(); +var era5_1980 = era5.filterDate('1981-01-01', '1991-01-01').mean(); +var era5_2010 = era5.filterDate('2011-01-01', '2020-01-01').mean(); // Compute the temperature change. -var era5_diff = era5_2010.subtract(era5_1980); +var era5_diff = era5_2010.subtract(era5_1980); // Show it on the map. Map.addLayer(era5_diff, { -   palette: threeColorPalette, -   min: -2, -   max: 2}, 'Blue White Red palette'); + palette: threeColorPalette, + min: -2, + max: 2}, 'Blue White Red palette'); Map.addLayer(era5_diff, { -   palette: balancePalette, -   min: -2, -   max: 2}, 'Balance palette'); + palette: balancePalette, + min: -2, + max: 2}, 'Balance palette'); -![](F6/image66.png)![](F6/image53.png) +``` +![Fig. F6.0.2 Temperature difference of ERA5 (2011–2020, 1981–1990) using the balance colormap from cmocean (right) versus a basic blue-white-red colormap (left)](F6/image66.png)![Fig. F6.0.2 Temperature difference of ERA5 (2011–2020, 1981–1990) using the balance colormap from cmocean (right) versus a basic blue-white-red colormap (left)](F6/image53.png) -Fig. F6.0.2 Temperature difference of ERA5 (2011–2020, 1981–1990) using the balance colormap from cmocean (right) versus a basic blue-white-red colormap (left) -::: {.callout-note} +:::{.callout-note} Code Checkpoint F60a. The book’s repository contains a script that shows what your code should look like at this point. ::: -Our second example in this section focuses on visualizing a region of the Antarctic, the Thwaites Glacier. This is one of the fast-flowing glaciers that causes concern because it loses so much mass that it causes the sea level to rise. If we want to visualize this region, we have a challenge. The Antarctic region is in the dark for four to five months each winter. That means that we can’t use optical images to see the ice flowing into the sea. We therefore will use radar images. Here we will use a semantic colormap to denote the meaning of the radar images. +Our second example in this section focuses on visualizing a region of the Antarctic, the Thwaites Glacier. This is one of the fast-flowing glaciers that causes concern because it loses so much mass that it causes the sea level to rise. If we want to visualize this region, we have a challenge. The Antarctic region is in the dark for four to five months each winter. That means that we can’t use optical images to see the ice flowing into the sea. We therefore will use radar images. Here we will use a semantic colormap to denote the meaning of the radar images. Let’s start by importing the dataset of radar images. We will use the images from the Sentinel-1 constellation of the Copernicus program. This satellite uses a C-band synthetic-aperture radar and has near-polar coverage. The radar senses images using a polarity for the sender and receiver. The collection has images of four different possible combinations of sender/receiver polarity pairs. The image that we’ll use has a band of the Horizontal/Horizontal polarity (HH). +```js // An image of the Thwaites glacier. -var imageId ='COPERNICUS/S1_GRD/S1B_EW_GRDM_1SSH_20211216T041925_20211216T042029_030045_03965B_AF0A'; +var imageId ='COPERNICUS/S1_GRD/S1B_EW_GRDM_1SSH_20211216T041925_20211216T042029_030045_03965B_AF0A'; // Look it up and select the HH band. -var img = ee.Image(imageId).select('HH'); +var img = ee.Image(imageId).select('HH'); -For the next step, we will use the palette library. We will stylize the radar images to look like optical images, so that viewers can contrast ice and sea ice from water (Lhermitte, 2020). We will use the Ice colormap from the cmocean package (Thyng, 2016). +``` +For the next step, we will use the palette library. We will stylize the radar images to look like optical images, so that viewers can contrast ice and sea ice from water (Lhermitte, 2020). We will use the Ice colormap from the cmocean package (Thyng, 2016). +```js // Use the palette library. -var palettes = require('users/gena/packages:palettes'); +var palettes = require('users/gena/packages:palettes'); // Access the ice palette. -var icePalette = palettes.cmocean.Ice[7]; +var icePalette = palettes.cmocean.Ice[7]; // Show it in the console. palettes.showPalette('Ice', icePalette); -// Use  it to visualize the radar data. +// Use it to visualize the radar data. Map.addLayer(img, { -   palette: icePalette, -   min: -15, -   max: 1}, 'Sentinel-1 radar'); + palette: icePalette, + min: -15, + max: 1}, 'Sentinel-1 radar'); // Zoom to the grounding line of the Thwaites Glacier. -Map.centerObject(ee.Geometry.Point([-105.45882094907664, -   74.90419580705336]), 8); +Map.centerObject(ee.Geometry.Point([-105.45882094907664, - 74.90419580705336]), 8); -If you zoom in (F6.0.3) you can see how long cracks have recently appeared near the pinning point (a peak in the bathymetry that functions as a buttress, see Wild, 2022) of the glacier. +``` +If you zoom in (F6.0.3) you can see how long cracks have recently appeared near the pinning point (a peak in the bathymetry that functions as a buttress, see Wild, 2022) of the glacier. -![](F6/image13.png) +![Fig. F6.0.3. Ice observed in Antarctica by the Sentinel-1 satellite. The image is rendered using the ice color palette stretched to backscatter amplitude values [-15; 1].](F6/image13.png) -Fig. F6.0.3. Ice observed in Antarctica by the Sentinel-1 satellite. The image is rendered using the ice color palette stretched to backscatter amplitude values [-15; 1]. -::: {.callout-note} +:::{.callout-note} Code Checkpoint F60b. The book’s repository contains a script that shows what your code should look like at this point. ::: -## Remapping and Palettes +## Remapping and Palettes -Classified rasters in Earth Engine have metadata attached that can help with analysis and visualization. This includes lists of the names, values, and colors associated with class. These are used as the default color palette for drawing a classification, as seen next. The USGS National Land Cover Database (NLCD) is one such example. Let’s access the NLCD dataset, name it nlcd, and view it (Fig. F6.0.4) with its built-in palette. +Classified rasters in Earth Engine have metadata attached that can help with analysis and visualization. This includes lists of the names, values, and colors associated with class. These are used as the default color palette for drawing a classification, as seen next. The USGS National Land Cover Database (NLCD) is one such example. Let’s access the NLCD dataset, name it nlcd, and view it (Fig. F6.0.4) with its built-in palette. +```js // Advanced remapping using NLCD. // Import NLCD. -var nlcd = ee.ImageCollection('USGS/NLCD_RELEASES/2016_REL'); +var nlcd = ee.ImageCollection('USGS/NLCD_RELEASES/2016_REL'); // Use Filter to select the 2016 dataset. -var nlcd2016 = nlcd.filter(ee.Filter.eq('system:index', '2016')) -   .first(); +var nlcd2016 = nlcd.filter(ee.Filter.eq('system:index', '2016')) + .first(); // Select the land cover band. -var landcover = nlcd2016.select('landcover'); +var landcover = nlcd2016.select('landcover'); // Map the NLCD land cover. Map.addLayer(landcover, null, 'NLCD Landcover'); -![](F6/image54.png) +``` +![Fig. F6.0.4 The NLCD visualized with default colors for each class](F6/image54.png) -Fig. F6.0.4 The NLCD visualized with default colors for each class But suppose you want to change the display palette. For example, you might want to have multiple classes displayed using the same color, or use different colors for some classes. Let’s try having all three urban classes display as dark red ('ab0000'). +```js // Now suppose we want to change the color palette. -var newPalette = ['466b9f', 'd1def8', 'dec5c5',   'ab0000', 'ab0000', 'ab0000',   'b3ac9f', '68ab5f', '1c5f2c',   'b5c58f', 'af963c', 'ccb879',   'dfdfc2', 'd1d182', 'a3cc51',   '82ba9e', 'dcd939', 'ab6c28',   'b8d9eb', '6c9fb8' +var newPalette = ['466b9f', 'd1def8', 'dec5c5', 'ab0000', 'ab0000', 'ab0000', 'b3ac9f', '68ab5f', '1c5f2c', 'b5c58f', 'af963c', 'ccb879', 'dfdfc2', 'd1d182', 'a3cc51', '82ba9e', 'dcd939', 'ab6c28', 'b8d9eb', '6c9fb8' ]; // Try mapping with the new color palette. Map.addLayer(landcover, { -   palette: newPalette + palette: newPalette }, 'NLCD New Palette'); +``` However, if you map this, you will see an unexpected result (Fig. F6.0.5). -![](F6/image64.png) +![Fig. F6.0.5 Applying a new palette to a multi-class layer has some unexpected results](F6/image64.png) -Fig. F6.0.5 Applying a new palette to a multi-class layer has some unexpected results This is because the numeric codes for the different classes are not sequential. Thus, Earth Engine stretches the given palette across the whole range of values and produces an unexpected color palette. To fix this issue, we will create a new index for the class values so that they are sequential. +```js // Extract the class values and save them as a list. -var values = ee.List(landcover.get('landcover_class_values')); +var values = ee.List(landcover.get('landcover_class_values')); // Print the class values to console. print('raw class values', values); // Determine the maximum index value -var maxIndex = values.size().subtract(1); +var maxIndex = values.size().subtract(1); // Create a new index for the remap -var indexes = ee.List.sequence(0, maxIndex); +var indexes = ee.List.sequence(0, maxIndex); // Print the updated class values to console. print('updated class values', indexes); // Remap NLCD and display it in the map. -var colorized = landcover.remap(values, indexes) -   .visualize({ -       min: 0, -       max: maxIndex, -       palette: newPalette -   }); +var colorized = landcover.remap(values, indexes) + .visualize({ + min: 0, + max: maxIndex, + palette: newPalette + }); Map.addLayer(colorized, {}, 'NLCD Remapped Colors'); +``` Using this remapping approach, we can properly visualize the new color palette (Fig. F6.0.6). -![](F6/image57.png) +![Fig. F6.0.6 Expected results of the new color palette. All urban areas are now correctly showing as dark red and the other land cover types remain their original color.](F6/image57.png) -Fig. F6.0.6 Expected results of the new color palette. All urban areas are now correctly showing as dark red and the other land cover types remain their original color. -::: {.callout-note} +:::{.callout-note} Code Checkpoint F60c. The book’s repository contains a script that shows what your code should look like at this point. ::: ## Annotations -Annotations are the way to visualize data on maps to provide additional information about raster values or any other data relevant to the context. In this case, this additional information is usually shown as geometries, text labels, diagrams, or other visual elements. Some annotations in Earth Engine can be added by making use of the ui portion of the Earth Engine API, resulting in graphical user interface elements such as labels or charts added on top of the map. However, it is frequently useful to render annotations as a part of images, such as by visualizing various image properties or to highlight specific areas. +Annotations are the way to visualize data on maps to provide additional information about raster values or any other data relevant to the context. In this case, this additional information is usually shown as geometries, text labels, diagrams, or other visual elements. Some annotations in Earth Engine can be added by making use of the ui portion of the Earth Engine API, resulting in graphical user interface elements such as labels or charts added on top of the map. However, it is frequently useful to render annotations as a part of images, such as by visualizing various image properties or to highlight specific areas. -In many cases, these annotations can be mixed with output images generated outside of Earth Engine, for example, by post-processing exported images using Python libraries or by annotating using GIS applications such as QGIS or ArcGIS. However, annotations could also be also very useful to highlight and/or label specific areas directly within the Code Editor. Earth Engine provides a sufficiently rich API to turn vector features and geometries into raster images which can serve as annotations. We recommend checking the ee.FeatureCollection.style function in the Earth Engine documentation to learn how geometries can be rendered. +In many cases, these annotations can be mixed with output images generated outside of Earth Engine, for example, by post-processing exported images using Python libraries or by annotating using GIS applications such as QGIS or ArcGIS. However, annotations could also be also very useful to highlight and/or label specific areas directly within the Code Editor. Earth Engine provides a sufficiently rich API to turn vector features and geometries into raster images which can serve as annotations. We recommend checking the ee.FeatureCollection.style function in the Earth Engine documentation to learn how geometries can be rendered. -For  textual annotation, we will make use of an external package 'users/gena/packages:text' that provides a way to render strings into raster images directly using the Earth Engine raster API. It is beyond the scope of the current tutorials to explain the implementation of this package, but internally this package makes use of bitmap fonts which are ingested into Earth Engine as raster assets and are used to turn every character of a provided string into image glyphs, which are then translated to desired coordinates. +For textual annotation, we will make use of an external package 'users/gena/packages:text' that provides a way to render strings into raster images directly using the Earth Engine raster API. It is beyond the scope of the current tutorials to explain the implementation of this package, but internally this package makes use of bitmap fonts which are ingested into Earth Engine as raster assets and are used to turn every character of a provided string into image glyphs, which are then translated to desired coordinates. -The API of the text package includes the following mandatory and optional arguments: +The API of the text package includes the following mandatory and optional arguments: /** * Draws a string as a raster image at a given point. @@ -269,139 +277,144 @@ The API of the text package includes the following mandatory and optional argum * @param {{string, Object}} options - optional properties used to style text * * The options dictionary may include one or more of the following: -*     fontSize       - 16|18|24|32 - the size of the font (default: 16) -*     fontType       - Arial|Consolas - the type of the font (default: Arial) -*     alignX         - left|center|right (default: left) -*     alignY         - top|center|bottom (default: top) -*     textColor      - text color string (default: ffffff - white) -*     textOpacity    - 0-1, opacity of the text (default: 0.9) -*     textWidth      - width of the text (default: 1) -*     outlineColor   - text outline color string (default: 000000 - black) -*     outlineOpacity - 0-1, opacity of the text outline (default: 0.4) -*     outlineWidth   - width of the text outlines (default: 0) +* fontSize - 16|18|24|32 - the size of the font (default: 16) +* fontType - Arial|Consolas - the type of the font (default: Arial) +* alignX - left|center|right (default: left) +* alignY - top|center|bottom (default: top) +* textColor - text color string (default: ffffff - white) +* textOpacity - 0-1, opacity of the text (default: 0.9) +* textWidth - width of the text (default: 1) +* outlineColor - text outline color string (default: 000000 - black) +* outlineOpacity - 0-1, opacity of the text outline (default: 0.4) +* outlineWidth - width of the text outlines (default: 0) */ -To demonstrate how to use this API, let’s render a simple 'Hello World!' text string placed at the map center using default text parameters. The code for this will be: +To demonstrate how to use this API, let’s render a simple 'Hello World!' text string placed at the map center using default text parameters. The code for this will be: +```js // Include the text package. -var text = require('users/gena/packages:text'); +var text = require('users/gena/packages:text'); // Configure map (change center and map type). Map.setCenter(0, 0, 10); Map.setOptions('HYBRID'); // Draw text string and add to map. -var pt = Map.getCenter(); -var scale = Map.getScale(); -var image = text.draw('Hello World!', pt, scale); +var pt = Map.getCenter(); +var scale = Map.getScale(); +var image = text.draw('Hello World!', pt, scale); Map.addLayer(image); -Running the above script will generate a new image containing the 'Hello World!' string placed in the map center. Notice that before calling the text.draw() function we configure the map to be centered at specific coordinates (0,0) and zoom level 10 because map parameters such as center and scale are passed as arguments to that text.draw() function. This ensures that the resulting image containing string characters is scaled properly. +``` +Running the above script will generate a new image containing the 'Hello World!' string placed in the map center. Notice that before calling the text.draw() function we configure the map to be centered at specific coordinates (0,0) and zoom level 10 because map parameters such as center and scale are passed as arguments to that text.draw() function. This ensures that the resulting image containing string characters is scaled properly. When exporting images containing rendered text strings, it is important to use proper scale to avoid distorted text strings that are difficult to read, depending on the selected font size, as shown in Fig. 6.0.7. -::: {.callout-note} +:::{.callout-note} Code Checkpoint F60d. The book’s repository contains a script that shows what your code should look like at this point. ::: -![](F6/image39.png)![](F6/image74.png)![](F6/image44.png) +![Fig. 6.0.7 Results of the text.draw call, scaled to 1x: var scale = Map.getScale()*1; (left), 2x: var scale = Map.getScale()*2; (center), and 0.5x: var scale = Map.getScale()*0.5; (right)](F6/image39.png)![Fig. 6.0.7 Results of the text.draw call, scaled to 1x: var scale = Map.getScale()*1; (left), 2x: var scale = Map.getScale()*2; (center), and 0.5x: var scale = Map.getScale()*0.5; (right)](F6/image74.png)![Fig. 6.0.7 Results of the text.draw call, scaled to 1x: var scale = Map.getScale()*1; (left), 2x: var scale = Map.getScale()*2; (center), and 0.5x: var scale = Map.getScale()*0.5; (right)](F6/image44.png) -Fig. 6.0.7 Results of the text.draw call, scaled to 1x: var scale = Map.getScale()*1; (left), 2x: var scale = Map.getScale()*2; (center), and 0.5x: var scale = Map.getScale()*0.5; (right) -These artifacts can be avoided to some extent by specifying a larger font size (e.g., 32). However, it is better to render text at the native 1:1 scale to achieve best results. The same applies to the text color and outline: They may need to be adjusted to achieve the best result. Usually, text needs to be rendered using colors that have opposite brightness and colors when compared to the surrounding background. Notice that in the above example, the map was configured to have a dark background ('HYBRID') to ensure that the white text (default color) would be visible. Multiple parameters listed in the above API documentation can be used to adjust text rendering. For example, let’s switch font size, font type, text, and outline parameters to render the same string, as below. Replace the existing one-line text.draw call in your script with the following code, and then run it again to see the difference (Fig. F6.0.8): +These artifacts can be avoided to some extent by specifying a larger font size (e.g., 32). However, it is better to render text at the native 1:1 scale to achieve best results. The same applies to the text color and outline: They may need to be adjusted to achieve the best result. Usually, text needs to be rendered using colors that have opposite brightness and colors when compared to the surrounding background. Notice that in the above example, the map was configured to have a dark background ('HYBRID') to ensure that the white text (default color) would be visible. Multiple parameters listed in the above API documentation can be used to adjust text rendering. For example, let’s switch font size, font type, text, and outline parameters to render the same string, as below. Replace the existing one-line text.draw call in your script with the following code, and then run it again to see the difference (Fig. F6.0.8): -var image = text.draw('Hello World!', pt, scale, { -   fontSize: 32, -   fontType: 'Consolas', -   textColor: 'black', -   outlineColor: 'white', -   outlineWidth: 1, -   outlineOpacity: 0.8 +var image = text.draw('Hello World!', pt, scale, { + fontSize: 32, + fontType: 'Consolas', + textColor: 'black', + outlineColor: 'white', + outlineWidth: 1, + outlineOpacity: 0.8 }); +```js // Add the text image to the map. Map.addLayer(image); -::: {.callout-note} +``` +:::{.callout-note} Code Checkpoint F60e. The book’s repository contains a script that shows what your code should look like at this point. ::: -![](F6/image33.png) +![Fig. 6.0.8 Rendering text with adjusted parameters (font type: Consolas, fontSize: 32, textColor: 'black', outlineWidth: 1, outlineColor: 'white', outlineOpacity: 0.8)](F6/image33.png) -Fig. 6.0.8 Rendering text with adjusted parameters (font type: Consolas, fontSize: 32, textColor: 'black', outlineWidth: 1, outlineColor: 'white', outlineOpacity: 0.8) -Of course, non-optional parameters such as pt and scale, as well as the text string, do not have to be hard-coded in the script; instead, they can be acquired by the code using, for example, properties coming from a FeatureCollection. Let's demonstrate this by showing the cloudiness of Landsat 8 images as text labels rendered in the center of every image. In addition to annotating every image with a cloudiness text string, we will also draw yellow outlines to indicate image boundaries. For convenience, we can also define the code to annotate an image as a function. We will then map that function (as described in Chap. F4.0) over the filtered ImageCollection. The code is as follows: +Of course, non-optional parameters such as pt and scale, as well as the text string, do not have to be hard-coded in the script; instead, they can be acquired by the code using, for example, properties coming from a FeatureCollection. Let's demonstrate this by showing the cloudiness of Landsat 8 images as text labels rendered in the center of every image. In addition to annotating every image with a cloudiness text string, we will also draw yellow outlines to indicate image boundaries. For convenience, we can also define the code to annotate an image as a function. We will then map that function (as described in Chap. F4.0) over the filtered ImageCollection. The code is as follows: -var text = require('users/gena/packages:text'); +var text = require('users/gena/packages:text'); -var geometry = ee.Geometry.Polygon( -   [ -       [ -           [-109.248, 43.3913], -           [-109.248, 33.2689], -           [-86.5283, 33.2689], -           [-86.5283, 43.3913] -       ] -   ], null, false); +var geometry = ee.Geometry.Polygon( + [ + [ + [-109.248, 43.3913], + [-109.248, 33.2689], + [-86.5283, 33.2689], + [-86.5283, 43.3913] + ] + ], null, false); Map.centerObject(geometry, 6); -function annotate(image) {   // Annotates an image by adding outline border and cloudiness   // Cloudiness is shown as a text string rendered at the image center.   // Add an edge around the image.   var edge = ee.FeatureCollection([image]) -       .style({ -           color: 'cccc00cc', -           fillColor: '00000000'       });   // Draw cloudiness as text.   var props = { -       textColor: '0000aa', -       outlineColor: 'ffffff', -       outlineWidth: 2, -       outlineOpacity: 0.6, -       fontSize: 24, -       fontType: 'Consolas'   };   var center = image.geometry().centroid(1);   var str = ee.Number(image.get('CLOUD_COVER')).format('%.2f');   var scale = Map.getScale();   var textCloudiness = text.draw(str, center, scale, props);   // Shift left 25 pixels.   textCloudiness = textCloudiness -       .translate(-scale * 25, 0, 'meters', 'EPSG:3857');   // Merge results.   return ee.ImageCollection([edge, textCloudiness]).mosaic(); +function annotate(image) { // Annotates an image by adding outline border and cloudiness // Cloudiness is shown as a text string rendered at the image center. // Add an edge around the image. var edge = ee.FeatureCollection([image]) + .style({ + color: 'cccc00cc', + fillColor: '00000000' }); // Draw cloudiness as text. var props = { + textColor: '0000aa', + outlineColor: 'ffffff', + outlineWidth: 2, + outlineOpacity: 0.6, + fontSize: 24, + fontType: 'Consolas' }; var center = image.geometry().centroid(1); var str = ee.Number(image.get('CLOUD_COVER')).format('%.2f'); var scale = Map.getScale(); var textCloudiness = text.draw(str, center, scale, props); // Shift left 25 pixels. textCloudiness = textCloudiness + .translate(-scale * 25, 0, 'meters', 'EPSG:3857'); // Merge results. return ee.ImageCollection([edge, textCloudiness]).mosaic(); } +```js // Select images. -var images = ee.ImageCollection('LANDSAT/LC08/C02/T1_RT_TOA') -   .select([5, 4, 2]) -   .filterBounds(geometry) -   .filterDate('2018-01-01', '2018-01-7'); +var images = ee.ImageCollection('LANDSAT/LC08/C02/T1_RT_TOA') + .select([5, 4, 2]) + .filterBounds(geometry) + .filterDate('2018-01-01', '2018-01-7'); // dim background. Map.addLayer(ee.Image(1), { -   palette: ['black'] + palette: ['black'] }, 'black', true, 0.5); // Show images. Map.addLayer(images, { -   min: 0.05, -   max: 1, -   gamma: 1.4}, 'images'); + min: 0.05, + max: 1, + gamma: 1.4}, 'images'); // Show annotations. -var labels = images.map(annotate); -var labelsLayer = ui.Map.Layer(labels, {}, 'annotations'); +var labels = images.map(annotate); +var labelsLayer = ui.Map.Layer(labels, {}, 'annotations'); Map.layers().add(labelsLayer); -The result of defining and mapping this function over the filtered set of images is shown in Fig. F6.0.9. Notice that by adding an outline around the text, we can ensure the text is visible for both dark and light images. Earth Engine requires casting properties to their corresponding value type, which is why we’ve used ee.Number (as described in Chap. F1.0) before generating a formatted string. Also, we have shifted the resulting text image 25 pixels to the left. This was necessary to ensure that the text is positioned properly. In more complex text rendering applications, users may be required to compute the text position in a different way using ee.Geometry calls from the Earth Engine API: for example, by positioning text labels somewhere near the corners. +``` +The result of defining and mapping this function over the filtered set of images is shown in Fig. F6.0.9. Notice that by adding an outline around the text, we can ensure the text is visible for both dark and light images. Earth Engine requires casting properties to their corresponding value type, which is why we’ve used ee.Number (as described in Chap. F1.0) before generating a formatted string. Also, we have shifted the resulting text image 25 pixels to the left. This was necessary to ensure that the text is positioned properly. In more complex text rendering applications, users may be required to compute the text position in a different way using ee.Geometry calls from the Earth Engine API: for example, by positioning text labels somewhere near the corners. -![](F6/image3.png) +![Fig. F6.0.9 Annotating Landsat 8 images with image boundaries, border, and text strings indicating cloudiness](F6/image3.png) -Fig. F6.0.9 Annotating Landsat 8 images with image boundaries, border, and text strings indicating cloudiness -Because we render text labels using the Earth Engine raster API, they are not automatically scaled depending on map zoom size. This may cause unwanted artifacts; To avoid that, the text labels image needs to be updated every time the map zoom changes. To implement this in a script, we can make use of the Map API—in particular, the Map.onChangeZoom event handler. The following code snippet shows how the image containing text annotations can be re-rendered every time the map zoom changes. Add it to the end of your script. +Because we render text labels using the Earth Engine raster API, they are not automatically scaled depending on map zoom size. This may cause unwanted artifacts; To avoid that, the text labels image needs to be updated every time the map zoom changes. To implement this in a script, we can make use of the Map API—in particular, the Map.onChangeZoom event handler. The following code snippet shows how the image containing text annotations can be re-rendered every time the map zoom changes. Add it to the end of your script. +```js // re-render (rescale) annotations when map zoom changes. Map.onChangeZoom(function(zoom) { -   labelsLayer.setEeObject(images.map(annotate)); + labelsLayer.setEeObject(images.map(annotate)); }); -::: {.callout-note} +``` +:::{.callout-note} Code Checkpoint F60f. The book’s repository contains a script that shows what your code should look like at this point. ::: Try commenting that event handler and observe how annotation rendering changes when you zoom in or zoom out. ## Animations -Visualizing raster images as animations is a useful technique to explore changes in time-dependent datasets, but also, to render short animations to communicate how changing various parameters affects the resulting image—for example, varying thresholds of spectral indices resulting in different binary maps or the changing geometry of vector features. +Visualizing raster images as animations is a useful technique to explore changes in time-dependent datasets, but also, to render short animations to communicate how changing various parameters affects the resulting image—for example, varying thresholds of spectral indices resulting in different binary maps or the changing geometry of vector features. -Animations are very useful when exploring satellite imagery, as they allow viewers to quickly comprehend dynamics of changes of earth surface or atmospheric properties. Animations can also help to decide what steps should be taken next to designing a robust algorithm to extract useful information from satellite image time series. Earth Engine provides two standard ways to generate animations: as animated GIFs, and as AVI video clips. Animation can also be rendered from a sequence of images exported from Earth Engine, using numerous tools such as ffmpeg or moviepy. However, in many cases it is useful to have a way to quickly explore image collections as animation without requiring extra steps. +Animations are very useful when exploring satellite imagery, as they allow viewers to quickly comprehend dynamics of changes of earth surface or atmospheric properties. Animations can also help to decide what steps should be taken next to designing a robust algorithm to extract useful information from satellite image time series. Earth Engine provides two standard ways to generate animations: as animated GIFs, and as AVI video clips. Animation can also be rendered from a sequence of images exported from Earth Engine, using numerous tools such as ffmpeg or moviepy. However, in many cases it is useful to have a way to quickly explore image collections as animation without requiring extra steps. In this section, we will generate animations in three different ways: @@ -411,25 +424,26 @@ In this section, we will generate animations in three different ways: We will use an image collection showing sea ice as an input dataset to generate animations with visualization parameters from earlier. However, instead of querying a single Sentinel-1 image, let’s generate a filtered image collection with all images intersecting with our area of interest. After importing some packages and palettes and defining a point and rectangle, we’ll build the image collection. Here we will use point geometry to define the location where the image date label will be rendered and the rectangle geometry to indicate the area of interest for the animation. To do this we will build the following logic in a new script. Open a new script and paste the following code into it: +```js // Include packages. -var palettes = require('users/gena/packages:palettes'); -var text = require('users/gena/packages:text'); +var palettes = require('users/gena/packages:palettes'); +var text = require('users/gena/packages:text'); -var point = /* color: #98ff00 */ ee.Geometry.Point([-   106.15944300895228, -74.58262940096245 +var point = /* color: #98ff00 */ ee.Geometry.Point([- 106.15944300895228, -74.58262940096245 ]); -var rect = /* color: #d63000 */   ee.Geometry.Polygon( -       [ -           [ -               [-106.19789515738981, -74.56509549360152], -               [-106.19789515738981, -74.78071448733921], -               [-104.98115931754606, -74.78071448733921], -               [-104.98115931754606, -74.56509549360152] -           ] -       ], null, false); +var rect = /* color: #d63000 */ ee.Geometry.Polygon( + [ + [ + [-106.19789515738981, -74.56509549360152], + [-106.19789515738981, -74.78071448733921], + [-104.98115931754606, -74.78071448733921], + [-104.98115931754606, -74.56509549360152] + ] + ], null, false); // Lookup the ice palette. -var palette = palettes.cmocean.Ice[7]; +var palette = palettes.cmocean.Ice[7]; // Show it in the console. palettes.showPalette('Ice', palette); @@ -438,70 +452,73 @@ palettes.showPalette('Ice', palette); Map.centerObject(point, 9); // Select S1 images for the Thwaites glacier. -var images = ee.ImageCollection('COPERNICUS/S1_GRD') -   .filterBounds(rect) -   .filterDate('2021-01-01', '2021-03-01') -   .select('HH')   // Make sure we include only images which fully contain the region geometry.   .filter(ee.Filter.isContained({ -       leftValue: rect, -       rightField: '.geo'   })) -   .sort('system:time_start'); +var images = ee.ImageCollection('COPERNICUS/S1_GRD') + .filterBounds(rect) + .filterDate('2021-01-01', '2021-03-01') + .select('HH') // Make sure we include only images which fully contain the region geometry. .filter(ee.Filter.isContained({ + leftValue: rect, + rightField: '.geo' })) + .sort('system:time_start'); // Print number of images. print(images.size()); -As you see from the last last lines of the above code, it is frequently useful to print the number of images in an image collection: an example of what’s often known as a “sanity check.” +``` +As you see from the last last lines of the above code, it is frequently useful to print the number of images in an image collection: an example of what’s often known as a “sanity check.” -Here we have used two custom geometries to configure animations: the green pin named point, used to filter image collection and to position text labels drawn on top of the image, and the blue rectangle rect, used to define a bounding box for the exported animations. To make sure that the point and rectangle geometries are shown under the Geometry Imports in the Code Editor, you need to click on these variables in the code and then select the Convert link. +Here we have used two custom geometries to configure animations: the green pin named point, used to filter image collection and to position text labels drawn on top of the image, and the blue rectangle rect, used to define a bounding box for the exported animations. To make sure that the point and rectangle geometries are shown under the Geometry Imports in the Code Editor, you need to click on these variables in the code and then select the Convert link. -Notice that in addition to the bounds and date filter, we have also used a less known isContained filter to ensure that we get only images that fully cover our region. To better understand this filter, you could try commenting out the filter and compare the differences, observing images with empty (masked) pixels in the resulting image collection. +Notice that in addition to the bounds and date filter, we have also used a less known isContained filter to ensure that we get only images that fully cover our region. To better understand this filter, you could try commenting out the filter and compare the differences, observing images with empty (masked) pixels in the resulting image collection. -::: {.callout-note} +:::{.callout-note} Code Checkpoint F60g. The book’s repository contains a script that shows what your code should look like at this point. ::: Next, to simplify the animation API calls, we will generate a composite RGB image collection out of satellite images and draw the image’s acquisition date as a label on every image, positioned within our region geometry. +```js // Render images. -var vis = { -   palette: palette, -   min: -15, -   max: 1 +var vis = { + palette: palette, + min: -15, + max: 1 }; -var scale = Map.getScale(); -var textProperties = { -   outlineColor: '000000', -   outlineWidth: 3, -   outlineOpacity: 0.6 +var scale = Map.getScale(); +var textProperties = { + outlineColor: '000000', + outlineWidth: 3, + outlineOpacity: 0.6 }; -var imagesRgb = images.map(function(i) {   // Use the date as the label.   var label = i.date().format('YYYY-MM-dd');   var labelImage = text.draw(label, point, scale, -       textProperties);   return i.visualize(vis) -       .blend(labelImage) // Blend label image on top.       .set({ -           label: label -       }); // Keep the text property. +var imagesRgb = images.map(function(i) { // Use the date as the label. var label = i.date().format('YYYY-MM-dd'); var labelImage = text.draw(label, point, scale, + textProperties); return i.visualize(vis) + .blend(labelImage) // Blend label image on top. .set({ + label: label + }); // Keep the text property. }); Map.addLayer(imagesRgb.first()); Map.addLayer(rect, {color:'blue'}, 'rect', 1, 0.5); -In addition to printing the size of the ImageCollection, we also often begin by adding a single image to the map from a mapped collection to see that everything works as expected—another example of a sanity check. The resulting map layer will look like F6.0.10. +``` +In addition to printing the size of the ImageCollection, we also often begin by adding a single image to the map from a mapped collection to see that everything works as expected—another example of a sanity check. The resulting map layer will look like F6.0.10. -![](F6/image6.png) +![Fig. F6.0.10 The results of adding the first layer from the RGB composite image collection showing Sentinel-1 images with a label blended on top at a specified location. The blue geometry is used to define the bounds for the animation to be exported.](F6/image6.png) -Fig. F6.0.10 The results of adding the first layer from the RGB composite image collection showing Sentinel-1 images with a label blended on top at a specified location. The blue geometry is used to define the bounds for the animation to be exported. -::: {.callout-note} +:::{.callout-note} Code Checkpoint F60h. The book’s repository contains a script that shows what your code should look like at this point. ::: Animation 1: Animated GIF with ui.Thumbnail The quickest way to generate an animation in Earth Engine is to use the animated GIF API and either print it to the Console or print the URL to download the generated GIF. The following code snippet will result in an animated GIF as well as the URL to the animated GIF printed to Console. This is as shown in Fig. F6.0.11: +```js // Define GIF visualization parameters. -var gifParams = { -   region: rect, -   dimensions: 600, -   crs: 'EPSG:3857', -   framesPerSecond: 10 +var gifParams = { + region: rect, + dimensions: 600, + crs: 'EPSG:3857', + framesPerSecond: 10 }; // Print the GIF URL to the console. @@ -510,40 +527,39 @@ print(imagesRgb.getVideoThumbURL(gifParams)); // Render the GIF animation in the console. print(ui.Thumbnail(imagesRgb, gifParams)); +``` Earth Engine provides multiple options to specify the size of the resulting video. In this example we specify 600 as the size of the maximum dimension. We also specify the number of frames per second for the resulting animated GIF as well as the target projected coordinate system to be used (EPSG:3857 here, which is the projection used in web maps such as Google Maps and the Code Editor background). -![](F6/image56.png) +![Fig. F6.0.11 Console output after running the animated GIF code snippet, showing the GIF URL and an animation shown directly in the Console](F6/image56.png) -Fig. F6.0.11 Console output after running the animated GIF code snippet, showing the GIF URL and an animation shown directly in the Console Animation 2: Exporting an Animation with Export.video.toDrive -Animated GIFs can be useful to generate animations quickly. However, they have several limitations. In particular, they are limited to 256 colors, become large for larger animations, and most web players do not provide play controls when playing animated GIFs. To overcome these limitations, Earth Engine provides export of animations as video files in MP4 format. Let’s use the same RGB image collection we have used for the animated GIF to generate a short video. We can ask Earth Engine to export the video to the Google Drive using the following code snippet: +Animated GIFs can be useful to generate animations quickly. However, they have several limitations. In particular, they are limited to 256 colors, become large for larger animations, and most web players do not provide play controls when playing animated GIFs. To overcome these limitations, Earth Engine provides export of animations as video files in MP4 format. Let’s use the same RGB image collection we have used for the animated GIF to generate a short video. We can ask Earth Engine to export the video to the Google Drive using the following code snippet: Export.video.toDrive({ -   collection: imagesRgb, -   description: 'ice-animation', -   fileNamePrefix: 'ice-animation', -   framesPerSecond: 10, -   dimensions: 600, -   region: rect, -   crs: 'EPSG:3857' + collection: imagesRgb, + description: 'ice-animation', + fileNamePrefix: 'ice-animation', + framesPerSecond: 10, + dimensions: 600, + region: rect, + crs: 'EPSG:3857' }); -Here, many arguments to the Export.video.toDrive function resemble the ones we’ve used in the ee.Image.getVideoThumbURL code above. Additional arguments include description and fileNamePrefix, which are required to configure the name of the task and the target file of the video file to be saved to Google Drive. Running the above code will result in a new task created under the Tasks tab in the Code Editor. Starting the export video task (F6.0.12) will result in a video file saved in the Google Drive once completed. +Here, many arguments to the Export.video.toDrive function resemble the ones we’ve used in the ee.Image.getVideoThumbURL code above. Additional arguments include description and fileNamePrefix, which are required to configure the name of the task and the target file of the video file to be saved to Google Drive. Running the above code will result in a new task created under the Tasks tab in the Code Editor. Starting the export video task (F6.0.12) will result in a video file saved in the Google Drive once completed. -![](F6/image9.png) +![Fig. F6.0.12 A new export video tasks in the Tasks panel of the Code Editor](F6/image9.png) -Fig. F6.0.12 A new export video tasks in the Tasks panel of the Code Editor Animation 3: The Custom Animation Package -For the last animation example, we will use the custom package 'users/gena/packages:animation', built using the Earth Engine User Interface API. The main difference between this package and the above examples is that it generates an interactive animation by adding Map layers individually to the layer set, and providing UI controls that allow users to play animations or interactively switch between frames. The animate function in that package generates an interactive animation of an ImageCollection, as described below. This function has a number of optional arguments allowing you to configure, for example, the number of frames to be animated, the number of frames to be preloaded, or a few others. The optional parameters to control the function are the following: +For the last animation example, we will use the custom package 'users/gena/packages:animation', built using the Earth Engine User Interface API. The main difference between this package and the above examples is that it generates an interactive animation by adding Map layers individually to the layer set, and providing UI controls that allow users to play animations or interactively switch between frames. The animate function in that package generates an interactive animation of an ImageCollection, as described below. This function has a number of optional arguments allowing you to configure, for example, the number of frames to be animated, the number of frames to be preloaded, or a few others. The optional parameters to control the function are the following: * maxFrames: maximum number of frames to show (default: 30) * vis: visualization parameters for every frame (default: {}) * Label: text property of images to show in the animation controls (default: undefined) -* width: width of the animation panel (default: '600px') +* width: width of the animation panel (default: '600px') * compact: show only play control and frame slider (default: false) * position: position of the animation panel (default: 'top-center') * timeStep: time step (ms) used when playing animation (default: 100) @@ -551,140 +567,143 @@ For the last animation example, we will use the custom package 'users/gena/packa Let’s call this function to add interactive animation controls to the current Map: +```js // include the animation package -var animation = require('users/gena/packages:animation'); +var animation = require('users/gena/packages:animation'); // show animation controls animation.animate(imagesRgb, { - label: 'label', - maxFrames: 50 + label: 'label', + maxFrames: 50 }); -Before using the interactive animation API, we need to include the corresponding package using require. Here we provide our pre-rendered image collection and two optional parameters (label and maxFrames). The first optional parameter label indicates that every image in our image collection has the 'label' text property. The animate function uses this property to name map layers as well as to visualize in the animation UI controls when switching between frames. This can be useful when inspecting image collections. The second optional parameter, maxFrames, indicates that the maximum number of animation frames that we would like to visualize is 50. To prevent the Code Editor from crashing, this parameter should not be too large: it is best to keep it below 100. For a much larger number of frames, it is better to use the Export video or animated GIF API. Running this code snippet will result in the animation control panel added to the map as shown in Fig. F6.0.13. +``` +Before using the interactive animation API, we need to include the corresponding package using require. Here we provide our pre-rendered image collection and two optional parameters (label and maxFrames). The first optional parameter label indicates that every image in our image collection has the 'label' text property. The animate function uses this property to name map layers as well as to visualize in the animation UI controls when switching between frames. This can be useful when inspecting image collections. The second optional parameter, maxFrames, indicates that the maximum number of animation frames that we would like to visualize is 50. To prevent the Code Editor from crashing, this parameter should not be too large: it is best to keep it below 100. For a much larger number of frames, it is better to use the Export video or animated GIF API. Running this code snippet will result in the animation control panel added to the map as shown in Fig. F6.0.13. It is important to note that the animation API uses asynchronous UI calls to make sure that the Code Editor does not hang when running the script. The drawback of this is that for complex image collections, a large amount of processing is required. Hence, it may take some time to process all images and to visualize the interactive animation panel. The same is true for map layer names: they are updated once the animation panel is visualized. Also, map layers used to visualize individual images in the provided image collection may require some time to be rendered. -![](F6/image70.png) +![Fig. F6.0.13 Interactive animation controls when using custom animation API](F6/image70.png) -Fig. F6.0.13 Interactive animation controls when using custom animation API The main advantage of the interactive animation API is that it provides a way to explore image collections at frame-by-frame basis, which can greatly improve our visual understanding of the changes captured in sets of images. -::: {.callout-note} +:::{.callout-note} Code Checkpoint F60i. The book’s repository contains a script that shows what your code should look like at this point. ::: ## Terrain Visualization This section introduces several raster visualization techniques useful to visualize terrain data such as: -* Basic hillshading and parameters (light azimuth, elevation) -* Combining elevation data and colors using HSV transform (Wikipedia: HSL and HSV) +* Basic hillshading and parameters (light azimuth, elevation) +* Combining elevation data and colors using HSV transform (Wikipedia: HSL and HSV) * Adding shadows -One special type of raster data is data that represents height. Elevation data can include topography, bathymetry, but also other forms of height, such as sea surface height can be presented as a terrain. +One special type of raster data is data that represents height. Elevation data can include topography, bathymetry, but also other forms of height, such as sea surface height can be presented as a terrain. Height is often visualized using the concept of directional light with a technique called hillshading. Because height is such a common feature in our environment, we also have an expectancy of how height is visualized. If height is visualized using a simple grayscale colormap, it looks very unnatural (Fig. F6.0.14, top left). By using hillshading, data immediately looks more natural (Fig. F6.0.14, top middle). -We can further improve the visualization by including shadows (Fig. F6.0.14, top right).  A final step is to replace the simple grayscale colormap with a perceptual uniform topographic colormap and mix this with the hillshading and shadows (Fig. F6.0.14, bottom). This section explains how to apply these techniques. +We can further improve the visualization by including shadows (Fig. F6.0.14, top right). A final step is to replace the simple grayscale colormap with a perceptual uniform topographic colormap and mix this with the hillshading and shadows (Fig. F6.0.14, bottom). This section explains how to apply these techniques. We’ll focus on elevation data stored in raster form. Elevation data is not always stored in raster formats. Other data formats include Triangulated Irregular Network (TIN), which allows storing information at varying resolutions and as 3D objects. This format allows one to have overlapping geometries, such as bridges with a road below it. In raster-based digital elevation models, in contrast, there can only be one height recorded for each pixel. Let’s start by loading data from a digital elevation model. This loads a topographic dataset from the Netherlands (Algemeen Hoogtebestand Nederland). It is a Digital Surface Model, based on airborne LIDAR measurements regridded to 0.5 m resolution. Enter the following code in a new script. -var dem = ee.Image('AHN/AHN2_05M_RUW'); +var dem = ee.Image('AHN/AHN2_05M_RUW'); -We can visualize this dataset using a sequential gradient colormap from black to white. This results in Fig. F6.0.14. One can infer which areas are lower and which are higher, but the visualization does not quite “feel” like a terrain. +We can visualize this dataset using a sequential gradient colormap from black to white. This results in Fig. F6.0.14. One can infer which areas are lower and which are higher, but the visualization does not quite “feel” like a terrain. +```js // Change map style to HYBRID and center map on the Netherlands Map.setOptions('HYBRID'); Map.setCenter(4.4082, 52.1775, 18); // Visualize DEM using black-white color palette -var palette = ['black', 'white']; -var demRGB = dem.visualize({ -   min: -5, -   max: 5, -   palette: palette +var palette = ['black', 'white']; +var demRGB = dem.visualize({ + min: -5, + max: 5, + palette: palette }); Map.addLayer(demRGB, {},'DEM'); -An important step to visualize terrain is to add shadows created by a distant point source of light. This is referred to as hillshading or a shaded relief map. This type of map became popular in the 1940s through the work of Edward Imhoff, who also used grayscale colormaps (Imhoff, 2015). Here we’ll use the 'gena/packages:utils' library to combine the colormap image with the shadows. That Earth Engine package implements a hillshadeRGB function to simplify rendering of images enhanced with hillshading and shadow effects. One important argument this function takes is the light azimuth—an angle from the image plane upward to the light source (the Sun). This should always be set to the top left to avoid bistable perception artifacts, in which the DEM can be misperceived as inverted. +``` +An important step to visualize terrain is to add shadows created by a distant point source of light. This is referred to as hillshading or a shaded relief map. This type of map became popular in the 1940s through the work of Edward Imhoff, who also used grayscale colormaps (Imhoff, 2015). Here we’ll use the 'gena/packages:utils' library to combine the colormap image with the shadows. That Earth Engine package implements a hillshadeRGB function to simplify rendering of images enhanced with hillshading and shadow effects. One important argument this function takes is the light azimuth—an angle from the image plane upward to the light source (the Sun). This should always be set to the top left to avoid bistable perception artifacts, in which the DEM can be misperceived as inverted. -var utils = require('users/gena/packages:utils'); +var utils = require('users/gena/packages:utils'); -var weight =   0.4; // Weight of Hillshade vs RGB (0 - flat, 1 - hillshaded). -var exaggeration = 5; // Vertical exaggeration. -var azimuth = 315; // Sun azimuth. -var zenith = 20; // Sun elevation. -var brightness = -0.05; // 0 - default. -var contrast = 0.05; // 0 - default. -var saturation = 0.8; // 1 - default. -var castShadows = false; +var weight = 0.4; // Weight of Hillshade vs RGB (0 - flat, 1 - hillshaded). +var exaggeration = 5; // Vertical exaggeration. +var azimuth = 315; // Sun azimuth. +var zenith = 20; // Sun elevation. +var brightness = -0.05; // 0 - default. +var contrast = 0.05; // 0 - default. +var saturation = 0.8; // 1 - default. +var castShadows = false; -var rgb = utils.hillshadeRGB( -   demRGB, dem, weight, exaggeration, azimuth, zenith, -   contrast, brightness, saturation, castShadows); +var rgb = utils.hillshadeRGB( + demRGB, dem, weight, exaggeration, azimuth, zenith, + contrast, brightness, saturation, castShadows); Map.addLayer(rgb, {}, 'DEM (no shadows)'); -Standard hillshading only determines per pixel if it will be directed to the light or not. One can also project shadows on the map. That is done using the ee.Algorithms.HillShadow algorithm. Here we’ll turn on castShadows in the hillshadeRGB function. This results in a more realistic map, as can be seen in Figure F6.0.14. +Standard hillshading only determines per pixel if it will be directed to the light or not. One can also project shadows on the map. That is done using the ee.Algorithms.HillShadow algorithm. Here we’ll turn on castShadows in the hillshadeRGB function. This results in a more realistic map, as can be seen in Figure F6.0.14. -var castShadows = true; +var castShadows = true; -var rgb = utils.hillshadeRGB( -   demRGB, dem, weight, exaggeration, azimuth, zenith, -   contrast, brightness, saturation, castShadows); +var rgb = utils.hillshadeRGB( + demRGB, dem, weight, exaggeration, azimuth, zenith, + contrast, brightness, saturation, castShadows); Map.addLayer(rgb, {}, 'DEM (with shadows)'); -The final step is to add a topographic colormap. To visualize topographic information, one often uses special topographic colormaps. Here we’ll use the oleron colormap from crameri. The colors get mixed with the shadows using the hillshadeRGB function. As you can see in Fig. F6.0.14, this gives a nice overview of the terrain. The area colored in blue is located below sea level. +The final step is to add a topographic colormap. To visualize topographic information, one often uses special topographic colormaps. Here we’ll use the oleron colormap from crameri. The colors get mixed with the shadows using the hillshadeRGB function. As you can see in Fig. F6.0.14, this gives a nice overview of the terrain. The area colored in blue is located below sea level. -var palettes = require('users/gena/packages:palettes'); -var palette = palettes.crameri.oleron[50]; +var palettes = require('users/gena/packages:palettes'); +var palette = palettes.crameri.oleron[50]; -var demRGB = dem.visualize({ -   min: -5, -   max: 5, -   palette: palette +var demRGB = dem.visualize({ + min: -5, + max: 5, + palette: palette }); -var castShadows = true; +var castShadows = true; -var rgb = utils.hillshadeRGB( -   demRGB, dem, weight, exaggeration, azimuth, zenith, -   contrast, brightness, saturation, castShadows); +var rgb = utils.hillshadeRGB( + demRGB, dem, weight, exaggeration, azimuth, zenith, + contrast, brightness, saturation, castShadows); Map.addLayer(rgb, {}, 'DEM colormap'); -Steps to further improve a terrain visualization include using light sources from multiple directions. This allows the user to render terrain to appear more natural. In the real world light is often scattered by clouds and other reflections. +Steps to further improve a terrain visualization include using light sources from multiple directions. This allows the user to render terrain to appear more natural. In the real world light is often scattered by clouds and other reflections. -One can also use lights to emphasize certain regions. To use even more advanced lighting techniques one can use a raytracing engine, such as the R rayshader library, as discussed earlier in this chapter. The raytracing engine in the Blender 3D program is also capable of producing stunning terrain visualizations using physical-based rendering, mist, environment lights, and camera effects such as depth of field. +One can also use lights to emphasize certain regions. To use even more advanced lighting techniques one can use a raytracing engine, such as the R rayshader library, as discussed earlier in this chapter. The raytracing engine in the Blender 3D program is also capable of producing stunning terrain visualizations using physical-based rendering, mist, environment lights, and camera effects such as depth of field. ![](F6/image36.png)![](F6/image58.png)![](F6/image47.png) ![](F6/image55.png) -Figure F6.0.14 Hillshading with shadows +Figure F6.0.14 Hillshading with shadows Steps in visualizing a topographic dataset: 1. Top left, topography with grayscale colormap 2. Top middle, topography with grayscale colormap and hillshading -3. Top right, topography with grayscale colormap, hillshading, and shadows +3. Top right, topography with grayscale colormap, hillshading, and shadows 4. Bottom, topography with topographic colormap, hillshading, and shadows -::: {.callout-note} -Code Checkpoint F60j. The book’s repository contains a script that shows what your code should look like at this point. +:::{.callout-note} +Code Checkpoint F60j. The book’s repository contains a script that shows what your code should look like at this point. ::: ## Synthesis {.unnumbered} To synthesize what you have learned in this chapter, you can do the following assignments. -Assignment 1. Experiment with different color palettes from the palettes library. Try combining palettes with image opacity (using ee.Image.updateMask call) to visualize different physical features (for example, hot or cold areas using temperature and elevation). +Assignment 1. Experiment with different color palettes from the palettes library. Try combining palettes with image opacity (using ee.Image.updateMask call) to visualize different physical features (for example, hot or cold areas using temperature and elevation). -Assignment 2. Render multiple text annotations when generating animations using image collection. For example, show other image properties in addition to date or image statistics generated using regional reducers for every image. +Assignment 2. Render multiple text annotations when generating animations using image collection. For example, show other image properties in addition to date or image statistics generated using regional reducers for every image. -Assignment 3. In addition to text annotations, try blending geometry elements (lines, polygons) to highlight specific areas of rendered images. +Assignment 3. In addition to text annotations, try blending geometry elements (lines, polygons) to highlight specific areas of rendered images. ## Conclusion {.unnumbered} @@ -712,13 +731,13 @@ Wilkinson L (2005) The Grammar of Graphics. Springer Verlag -# Collaborating in Earth Engine with Scripts and Assets +# Collaborating in Earth Engine with Scripts and Assets -::: {.callout-tip} +:::{.callout-tip} # Chapter Information ## Author {.unlisted .unnumbered} @@ -745,7 +764,7 @@ Many users find themselves needing to collaborate with others in Earth Engine at * Sharing a repository with others. * Seeing who made changes to a script and what changes were made. * Reverting to a previous version of a script. -* Using the require function to load modules. +* Using the require function to load modules. * Creating a script to share as a module. ## Assumes you know how to:{.unlisted .unnumbered} @@ -753,17 +772,17 @@ Many users find themselves needing to collaborate with others in Earth Engine at * Sign up for an Earth Engine account, open the Code Editor, and save your script (Chap. F1.0). -::: +::: ## Introduction {.unlisted .unnumbered} Many people find themselves needing to share a script when they encounter a problem; they wish to share the script with someone else so they can ask a question. When this occurs, sharing a link to the script often suffices. The other person can then make comments or changes before sending a new link to the modified script. -If you have included any assets from the Asset Manager in your script, you will also need to share these assets in order for your script to work for your colleague. The same goes for sharing assets to be displayed in an app. +If you have included any assets from the Asset Manager in your script, you will also need to share these assets in order for your script to work for your colleague. The same goes for sharing assets to be displayed in an app. Another common situation involves collaborating with others on a project. They may have some scripts they have written that they want to reuse or modify for the new project. Alternatively, several people might want to work on the same script together. For this situation, sharing a repository would be the best way forward; team members will be able to see who made what changes to a script and even revert to a previous version. -If you or your group members find yourselves repeatedly reusing certain functions for visualization or for part of your analysis, you could use the require module to call that function instead of having to copy and paste it into a new script each time. You could even make this function or module available to others to use via require. +If you or your group members find yourselves repeatedly reusing certain functions for visualization or for part of your analysis, you could use the require module to call that function instead of having to copy and paste it into a new script each time. You could even make this function or module available to others to use via require. Let’s get started. For this lab, you will need to work in small groups or pairs. @@ -774,309 +793,306 @@ Copy and paste the following code into the Code Editor. print('The author of this script is MyName.'); -Replace MyName with your name, then click on Save to save the script in your home repository. Next, click on the Get Link button and copy the link to this script onto your clipboard. Using your email program of choice, send this script to one of your group members. +Replace MyName with your name, then click on Save to save the script in your home repository. Next, click on the Get Link button and copy the link to this script onto your clipboard. Using your email program of choice, send this script to one of your group members. -Now add the following code below the line of code that you pasted earlier. +Now add the following code below the line of code that you pasted earlier. print('I just sent this script to GroupMemberName.'); -Replace GroupMemberName with the name of the person you sent this script to, then save the script again. Next, click on the Get Link button and copy the link to this script onto your clipboard. Using your email program of choice, send this script to the same person. +Replace GroupMemberName with the name of the person you sent this script to, then save the script again. Next, click on the Get Link button and copy the link to this script onto your clipboard. Using your email program of choice, send this script to the same person. Question 1. You should also have received two emails from someone in your group who is also doing this exercise. Open the first and second links in your Code Editor by clicking on them. Is the content of both scripts the same? -Answer: No, the scripts will be different, because Get Link sends a snapshot of the script at a particular point in time. Thus, even though the script was updated, the first link does not reflect that change. +Answer: No, the scripts will be different, because Get Link sends a snapshot of the script at a particular point in time. Thus, even though the script was updated, the first link does not reflect that change. -Question 2. What happens when you check the box for Hide code panel or Disable auto-run before sharing the script? +Question 2. What happens when you check the box for Hide code panel or Disable auto-run before sharing the script? -Answer. Hide code panel will minimize the code panel so the person you send the script to will see the Map maximized. This is useful when you want to draw the person’s attention to the results rather than to the code. To expand the code panel, they have to click on the Show code button. Disable auto-run is helpful when you do not want the script to start running when the person you sent it to opens it. Perhaps your script takes very long to run or requires particular user inputs and you just want to share the code with the person. +Answer. Hide code panel will minimize the code panel so the person you send the script to will see the Map maximized. This is useful when you want to draw the person’s attention to the results rather than to the code. To expand the code panel, they have to click on the Show code button. Disable auto-run is helpful when you do not want the script to start running when the person you sent it to opens it. Perhaps your script takes very long to run or requires particular user inputs and you just want to share the code with the person. ## Sharing Assets from Your Asset Manager -When you clicked the Get Link button earlier, you may have noticed a note in the popup reading: “To give others access to assets in the code snapshot, you may need to share them.” If your script uses an asset that you have uploaded into your Asset Manager, you will need to share that asset as well. If not, an error message will appear when the person you shared the script with tries to run it. +When you clicked the Get Link button earlier, you may have noticed a note in the popup reading: “To give others access to assets in the code snapshot, you may need to share them.” If your script uses an asset that you have uploaded into your Asset Manager, you will need to share that asset as well. If not, an error message will appear when the person you shared the script with tries to run it. Before sharing an asset, think about whether you have permission to share it. Is this some data that is owned by you, or did you get it from somewhere else? Do you need permission to share this asset? Make sure you have the permission to share an asset before doing so. -Now, let’s practice sharing assets. First, navigate to your Asset Manager by clicking on the Assets tab in the left panel. If you already have some assets uploaded, pick one that you have permission to share. If not, upload one to your Asset Manager. If you don’t have a shapefile or raster to upload, you can upload a small text file. Consult the Earth Engine documentation for how to do this; it will take only a few steps. +Now, let’s practice sharing assets. First, navigate to your Asset Manager by clicking on the Assets tab in the left panel. If you already have some assets uploaded, pick one that you have permission to share. If not, upload one to your Asset Manager. If you don’t have a shapefile or raster to upload, you can upload a small text file. Consult the Earth Engine documentation for how to do this; it will take only a few steps. -Hover your cursor over that asset in your Asset Manager. The asset gets highlighted in gray and three buttons appear to the right of the asset. Click on the first button from the left (outlined in red in Fig. F6.1.1). This icon means “share.” +Hover your cursor over that asset in your Asset Manager. The asset gets highlighted in gray and three buttons appear to the right of the asset. Click on the first button from the left (outlined in red in Fig. F6.1.1). This icon means “share.” -![](F6/image26.png) +![Fig. F6.1.1 Three assets in the Asset Manager](F6/image26.png) -Fig. F6.1.1 Three assets in the Asset Manager -After you click the share button, a Share Image popup will appear (Fig. F6.1.2). This popup contains information about the path of the asset and the email address of the owner. The owner of the asset can decide who can view and edit the asset. +After you click the share button, a Share Image popup will appear (Fig. F6.1.2). This popup contains information about the path of the asset and the email address of the owner. The owner of the asset can decide who can view and edit the asset. -Click on the dropdown menu outlined in red in Fig. F6.1.2. You will see two options for permissions: Reader and Writer. A Reader can view the asset, while a Writer can both view and make changes to it. For example, a Writer could add a new image to an ImageCollection. A Writer can also add other people to view or edit the asset, and a Writer can delete the asset. When in doubt, give someone the Reader role rather than the Writer role. +Click on the dropdown menu outlined in red in Fig. F6.1.2. You will see two options for permissions: Reader and Writer. A Reader can view the asset, while a Writer can both view and make changes to it. For example, a Writer could add a new image to an ImageCollection. A Writer can also add other people to view or edit the asset, and a Writer can delete the asset. When in doubt, give someone the Reader role rather than the Writer role. -![](F6/image10.png) +![Fig. F6.1.2 The Share Image popup window](F6/image10.png) -Fig. F6.1.2 The Share Image popup window -To share an asset with someone, you can type their email address into the Email or domain text field, choose Reader or Writer in the dropdown menu, and then click on Add Access. You can also share an asset with everyone with a certain email domain, which is useful if you want to share an asset with everyone in your organization, for instance. +To share an asset with someone, you can type their email address into the Email or domain text field, choose Reader or Writer in the dropdown menu, and then click on Add Access. You can also share an asset with everyone with a certain email domain, which is useful if you want to share an asset with everyone in your organization, for instance. -If you want to share reading access publicly, then check the box that says Anyone can read. Note that you still need to share the link to the asset in order for others to access it. The only exceptions to this are when you are using the asset in a script and sharing that script using the Get Link button or when you share the asset with an Earth Engine app. To do the latter, use the Select an app dropdown menu (outlined in orange in Fig. F6.1.2) and click Add App Access. +If you want to share reading access publicly, then check the box that says Anyone can read. Note that you still need to share the link to the asset in order for others to access it. The only exceptions to this are when you are using the asset in a script and sharing that script using the Get Link button or when you share the asset with an Earth Engine app. To do the latter, use the Select an app dropdown menu (outlined in orange in Fig. F6.1.2) and click Add App Access. Question 3. Share an asset with a group member and give them reader access. Send them the link to that asset. You will also receive a link from someone else in your group. Open that link. What can you do with that asset? What do you need to do to import it into a script? -Answer: You can view details about the asset and import it for use in a script in the Code Editor. To import the asset, click on the blue Import button. +Answer: You can view details about the asset and import it for use in a script in the Code Editor. To import the asset, click on the blue Import button. Question 4. Share an asset with a group member and give them writer access. Send them the link to that asset. You will also receive a link from someone else in your group. Open that link. What can you do with that asset? Try sharing the asset with a different group member. -Answer: You can view details about the asset and import it for use in a script in the Code Editor. You can also share the asset with others and delete the asset. +Answer: You can view details about the asset and import it for use in a script in the Code Editor. You can also share the asset with others and delete the asset. ## Working with Shared Repositories Now that you know how to share assets and scripts, let’s move on to sharing repositories. In this section, you will learn about different types of repositories and how to add a repository that someone else shared with you. You will also learn how to view previous versions of a script and how to revert back to an earlier version. -Earlier, we learned how to share a script using the Get Link button. This link shares a code snapshot from a script. This snapshot does not reflect any changes made to the script after the time the link was shared. If you want to share a script that updates to reflect the most current version when it is opened, you need to share a repository with that script instead. +Earlier, we learned how to share a script using the Get Link button. This link shares a code snapshot from a script. This snapshot does not reflect any changes made to the script after the time the link was shared. If you want to share a script that updates to reflect the most current version when it is opened, you need to share a repository with that script instead. -If you look under the Scripts tab of the leftmost panel in the Code Editor, you will see that the first three categories are labeled Owner, Reader, and Writer. +If you look under the Scripts tab of the leftmost panel in the Code Editor, you will see that the first three categories are labeled Owner, Reader, and Writer. -* Repositories categorized under Owner are created and owned by you. No one else has access to view or make changes to them until you share these repositories. -* Repositories categorized under Reader are repositories to which you have reader access. You can view the scripts but not make any changes to them. If you want to make any changes, you will need to save the script as a new file in a repository that you own. -* Repositories categorized under Writer are repositories to which you have writer access. This means you can view and make changes to the scripts. +* Repositories categorized under Owner are created and owned by you. No one else has access to view or make changes to them until you share these repositories. +* Repositories categorized under Reader are repositories to which you have reader access. You can view the scripts but not make any changes to them. If you want to make any changes, you will need to save the script as a new file in a repository that you own. +* Repositories categorized under Writer are repositories to which you have writer access. This means you can view and make changes to the scripts. -Let’s practice creating and sharing repositories. We will start by making a new repository. Click on the red New button located in the left panel. Select Repository from the dropdown menu. A New repository popup window will open (Fig. F6.1.3). +Let’s practice creating and sharing repositories. We will start by making a new repository. Click on the red New button located in the left panel. Select Repository from the dropdown menu. A New repository popup window will open (Fig. F6.1.3). -![](F6/image25.png) +![Fig. F6.1.3 The New repository popup window](F6/image25.png) -Fig. F6.1.3 The New repository popup window -In the popup window’s text field, type a name for your new repository, such as “ForSharing1,” then click on the blue Create button. You will see the new repository appear under the Owner category in the Scripts tab (Fig. F6.1.4). +In the popup window’s text field, type a name for your new repository, such as “ForSharing1,” then click on the blue Create button. You will see the new repository appear under the Owner category in the Scripts tab (Fig. F6.1.4). -Now, share this new repository with your group members: Hover your cursor over the repository you want to share. The repository gets highlighted in gray, and three buttons appear. Click on the Gear icon (outlined in red in Fig. F6.1.4). +Now, share this new repository with your group members: Hover your cursor over the repository you want to share. The repository gets highlighted in gray, and three buttons appear. Click on the Gear icon (outlined in red in Fig. F6.1.4). -![](F6/image71.png) +![Fig. F6.1.4 Three repositories under the Owner category](F6/image71.png) -Fig. F6.1.4 Three repositories under the Owner category -A Share Repo popup window appears (Fig. F6.1.5) which is very similar to the Share Image popup window we saw in Fig. F6.1.2. The method for sharing a repository with a specific user or the general public is the same as for sharing assets. +A Share Repo popup window appears (Fig. F6.1.5) which is very similar to the Share Image popup window we saw in Fig. F6.1.2. The method for sharing a repository with a specific user or the general public is the same as for sharing assets. -Type the email address of a group member in the Email or domain text field and give this person a writer role by selecting Writer in the dropdown menu, then click on Add Access.  +Type the email address of a group member in the Email or domain text field and give this person a writer role by selecting Writer in the dropdown menu, then click on Add Access. -![](F6/image29.png) +![Fig. F6.1.5. The Share Repo popup window](F6/image29.png) -Fig. F6.1.5. The Share Repo popup window Your group member should receive an email inviting them to edit the repository. Check your email inbox for the repository that your group member has shared with you. When you open that email, you will see content similar to what is shown in Fig. F6.1.6. -![](F6/image50.png) +![Fig. F6.1.6 The “Invitation to edit” email](F6/image50.png) -Fig. F6.1.6 The “Invitation to edit” email -Now, click on the blue button that says Add [repository path] to your Earth Engine Code Editor. You will find the new repository added to the Writer category in your Scripts tab. The repository path will contain the username of your group member, such as users/username/sharing. +Now, click on the blue button that says Add [repository path] to your Earth Engine Code Editor. You will find the new repository added to the Writer category in your Scripts tab. The repository path will contain the username of your group member, such as users/username/sharing. -Now, let’s add a script to the empty repository. Click on the red New button in the Scripts tab and select File from the dropdown menu. A Create file popup will appear, as shown in Fig. F6.1.7. Click on the gray arrow beside the default path to open a dropdown menu that will allow you to choose the path of the repository that your group member shared with you. Type a new File Name in the text field, such as “exercise,” then click on the blue OK button to create the file. +Now, let’s add a script to the empty repository. Click on the red New button in the Scripts tab and select File from the dropdown menu. A Create file popup will appear, as shown in Fig. F6.1.7. Click on the gray arrow beside the default path to open a dropdown menu that will allow you to choose the path of the repository that your group member shared with you. Type a new File Name in the text field, such as “exercise,” then click on the blue OK button to create the file. -![](F6/image69.png) +![Fig. F6.1.7 The Create file popup window](F6/image69.png) -Fig. F6.1.7 The Create file popup window -A new file should now appear in the shared repository in the Writer category. If you don’t see it, click on the Refresh icon, which is to the right of the red New button in the Scripts tab. +A new file should now appear in the shared repository in the Writer category. If you don’t see it, click on the Refresh icon, which is to the right of the red New button in the Scripts tab. Double-click on the new script in the shared repository to open it. Then, copy and paste the following code to your Code Editor. print('The owner of this repository is GroupMemberName.'); -Replace GroupMemberName with the name of your group member, then click Save to save the script in the shared repository, which is under the Writer category. +Replace GroupMemberName with the name of your group member, then click Save to save the script in the shared repository, which is under the Writer category. -Now, navigate to the repository under Owner which you shared with your group member. Open the new script which they just created by double-clicking it. +Now, navigate to the repository under Owner which you shared with your group member. Open the new script which they just created by double-clicking it. -Add the following code below the line of code that you pasted earlier. +Add the following code below the line of code that you pasted earlier. print('This script is shared with MyName.'); -Replace MyName with your name, then save the script. +Replace MyName with your name, then save the script. -Next, we will compare changes made to the script. Click on the Versions icon (outlined in red in Fig. F6.1.8). +Next, we will compare changes made to the script. Click on the Versions icon (outlined in red in Fig. F6.1.8). -![](F6/image17.png) +![Fig. F6.1.8 Changes made and previous versions of the script](F6/image17.png) -Fig. F6.1.8 Changes made and previous versions of the script -A popup window will appear, titled Revision history, followed by the path of the script (Fig. F6.1.9). There are three columns of information below the title. +A popup window will appear, titled Revision history, followed by the path of the script (Fig. F6.1.9). There are three columns of information below the title. -* The left column contains the dates on which changes have  been made. +* The left column contains the dates on which changes have been made. * The middle column contains the usernames of the people who made changes. -* The right column contains information about what changes were made.   +* The right column contains information about what changes were made. The most recent version of the script is shown in the first row, while previous versions are listed in subsequent rows. (More advanced users may notice that this is actually a Git repository.) -![](F6/image19.png) +![Fig. F6.1.9 The Revision history popup window](F6/image19.png) -Fig. F6.1.9 The Revision history popup window -If you hover your cursor over a row, the row will be highlighted in gray and a button labeled Compare will appear. Clicking on this button allows you to compare differences between the current version of the script and a previous version in a Version comparison popup window (Fig. F6.1.10). +If you hover your cursor over a row, the row will be highlighted in gray and a button labeled Compare will appear. Clicking on this button allows you to compare differences between the current version of the script and a previous version in a Version comparison popup window (Fig. F6.1.10). -![](F6/image30.png) +![Fig. F6.1.10 The Version comparison popup window](F6/image30.png) -Fig. F6.1.10 The Version comparison popup window -In the Version comparison popup, you will see text highlighted in two different colors. Text highlighted in red shows code that was present in the older version but is absent in the current version (the “latest commit”). Text highlighted in green shows code that is present in the current version but that was absent in the older version. Generally speaking, text highlighted in red has been removed in the current version and text highlighted in green has been added to the current version. Text that is not highlighted shows code that is present in both versions. +In the Version comparison popup, you will see text highlighted in two different colors. Text highlighted in red shows code that was present in the older version but is absent in the current version (the “latest commit”). Text highlighted in green shows code that is present in the current version but that was absent in the older version. Generally speaking, text highlighted in red has been removed in the current version and text highlighted in green has been added to the current version. Text that is not highlighted shows code that is present in both versions. -Question 5. What text, if any, is highlighted in red when you click on Compare in your “exercise” script? +Question 5. What text, if any, is highlighted in red when you click on Compare in your “exercise” script? -Answer: No text is highlighted in red, because none was removed between the previous and current versions of the script. +Answer: No text is highlighted in red, because none was removed between the previous and current versions of the script. -Question 6. What text, if any, is highlighted in green when you click on Compare in your “exercise” script? +Question 6. What text, if any, is highlighted in green when you click on Compare in your “exercise” script? -Answer: print('This script is shared with MyName.'); +Answer: print('This script is shared with MyName.'); -Question 7. What happens when you click on the blue Revert button? +Question 7. What happens when you click on the blue Revert button? -Answer: The script reverts to the previous version, in which the only line of code is +Answer: The script reverts to the previous version, in which the only line of code is print('The owner of this repository is GroupMemberName.'); ## Using the Require Function to Load a Module -In earlier chapters, you may have noticed that the require function allows you to reuse code that has already been written without having to copy and paste it into your current script. For example, you might have written a function for cloud masking that you would like to use in multiple scripts. Saving this function as a module enables you to share the code across your own scripts and with other people. Or you might discover a new module with capabilities you need written by other authors. This section will show you how to use the require function to create and share your own module or to load a module that someone else has shared. +In earlier chapters, you may have noticed that the require function allows you to reuse code that has already been written without having to copy and paste it into your current script. For example, you might have written a function for cloud masking that you would like to use in multiple scripts. Saving this function as a module enables you to share the code across your own scripts and with other people. Or you might discover a new module with capabilities you need written by other authors. This section will show you how to use the require function to create and share your own module or to load a module that someone else has shared. -The module we will use is ee-palettes, which enables users to visualize raster data using common specialized color palettes (Donchyts et al. 2019). (If you would like to learn more about using these color palettes, the ee-palettes module is described and illustrated in detail in Chap. F6.0.) The first step is to go to this link to accept access to the repository as a reader: [https://code.earthengine.google.com/?accept_repo=users/gena/](https://www.google.com/url?q=https://code.earthengine.google.com/?accept_repo%3Dusers/gena/packages&sa=D&source=editors&ust=1671458841147867&usg=AOvVaw2lfbVvfKSe6Nym_B5h25Z7)[packages](https://www.google.com/url?q=https://code.earthengine.google.com/?accept_repo%3Dusers/gena/packages&sa=D&source=editors&ust=1671458841148247&usg=AOvVaw396ktWqHZ6LRi90IQjOwxZ)  +The module we will use is ee-palettes, which enables users to visualize raster data using common specialized color palettes (Donchyts et al. 2019). (If you would like to learn more about using these color palettes, the ee-palettes module is described and illustrated in detail in Chap. F6.0.) The first step is to go to this link to accept access to the repository as a reader: [https://code.earthengine.google.com/?accept_repo=users/gena/](https://www.google.com/url?q=https://code.earthengine.google.com/?accept_repo%3Dusers/gena/packages&sa=D&source=editors&ust=1671458841147867&usg=AOvVaw2lfbVvfKSe6Nym_B5h25Z7)[packages](https://www.google.com/url?q=https://code.earthengine.google.com/?accept_repo%3Dusers/gena/packages&sa=D&source=editors&ust=1671458841148247&usg=AOvVaw396ktWqHZ6LRi90IQjOwxZ) -Now, if you navigate to your Reader directory in the Code Editor, you should see a new repository called 'users/gena/packages' listed. Look for a script called 'palettes' and click on it to load it in your Code Editor. +Now, if you navigate to your Reader directory in the Code Editor, you should see a new repository called 'users/gena/packages' listed. Look for a script called 'palettes' and click on it to load it in your Code Editor. -If you scroll down, you will see that the script contains a nested series of dictionaries with lists of hexadecimal color specifications (as described in Chap. F2.1) that describe a color palette, as shown in the code block below. For example, the color palette named “Algae” stored in the cmocean variable consists of seven colors, ranging from dark green to light green (Fig. F6.1.11). +If you scroll down, you will see that the script contains a nested series of dictionaries with lists of hexadecimal color specifications (as described in Chap. F2.1) that describe a color palette, as shown in the code block below. For example, the color palette named “Algae” stored in the cmocean variable consists of seven colors, ranging from dark green to light green (Fig. F6.1.11). -exports.cmocean = { -   Thermal: {       7: ['042333', '2c3395', '744992', 'b15f82', 'eb7958',           'fbb43d', 'e8fa5b'       ] -   }, -   Haline: {       7: ['2a186c', '14439c', '206e8b', '3c9387', '5ab978',           'aad85c', 'fdef9a'       ] -   }, -   Solar: {       7: ['331418', '682325', '973b1c', 'b66413', 'cb921a',           'dac62f', 'e1fd4b'       ] -   }, -   Ice: {       7: ['040613', '292851', '3f4b96', '427bb7', '61a8c7',           '9cd4da', 'eafdfd'       ] -   }, -   Gray: {       7: ['000000', '232323', '4a4a49', '727171', '9b9a9a',           'cacac9', 'fffffd'       ] -   }, -   Oxy: {       7: ['400505', '850a0b', '6f6f6e', '9b9a9a', 'cbcac9',           'ebf34b', 'ddaf19'       ] -   }, -   Deep: {       7: ['fdfecc', 'a5dfa7', '5dbaa4', '488e9e', '3e6495',           '3f396c', '281a2c'       ] -   }, -   Dense: {       7: ['e6f1f1', 'a2cee2', '76a4e5', '7871d5', '7642a5',           '621d62', '360e24'       ] -   }, -   Algae: {       7: ['d7f9d0', 'a2d595', '64b463', '129450', '126e45',           '1a482f', '122414'       ] -   }, -   ... +exports.cmocean = { + Thermal: { 7: ['042333', '2c3395', '744992', 'b15f82', 'eb7958', 'fbb43d', 'e8fa5b' ] + }, + Haline: { 7: ['2a186c', '14439c', '206e8b', '3c9387', '5ab978', 'aad85c', 'fdef9a' ] + }, + Solar: { 7: ['331418', '682325', '973b1c', 'b66413', 'cb921a', 'dac62f', 'e1fd4b' ] + }, + Ice: { 7: ['040613', '292851', '3f4b96', '427bb7', '61a8c7', '9cd4da', 'eafdfd' ] + }, + Gray: { 7: ['000000', '232323', '4a4a49', '727171', '9b9a9a', 'cacac9', 'fffffd' ] + }, + Oxy: { 7: ['400505', '850a0b', '6f6f6e', '9b9a9a', 'cbcac9', 'ebf34b', 'ddaf19' ] + }, + Deep: { 7: ['fdfecc', 'a5dfa7', '5dbaa4', '488e9e', '3e6495', '3f396c', '281a2c' ] + }, + Dense: { 7: ['e6f1f1', 'a2cee2', '76a4e5', '7871d5', '7642a5', '621d62', '360e24' ] + }, + Algae: { 7: ['d7f9d0', 'a2d595', '64b463', '129450', '126e45', '1a482f', '122414' ] + }, + ... } -Notice that the variable is named exports.cmocean. Adding exports to the name of a function or variable makes it available to other scripts to use, as it gets added to a special global variable (Chang 2017). +Notice that the variable is named exports.cmocean. Adding exports to the name of a function or variable makes it available to other scripts to use, as it gets added to a special global variable (Chang 2017). -![](F6/image52.png) +![Fig. F6.1.11 Some of the color palettes from the ee-palettes GitHub repository](F6/image52.png) -Fig. F6.1.11 Some of the color palettes from the ee-palettes GitHub repository To see all the color palettes available in this module, go to [https://github.com/gee-community/ee-palettes](https://www.google.com/url?q=https://github.com/gee-community/ee-palettes&sa=D&source=editors&ust=1671458841155957&usg=AOvVaw30bltn2S4_BDlhyuKIvkZH). -Now let’s try using the ee-palettes module. Look for a script in the same repository called 'palettes-test' and click on it to load it in your Code Editor. When you run the script, you will see digital elevation data from the National Aeronautics and Space Administration Shuttle Radar Topography Mission satellite visualized using two palettes,  colorbrewer.Blues and cmocean.Algae. The map will have two layers that show the same data with different palettes. +Now let’s try using the ee-palettes module. Look for a script in the same repository called 'palettes-test' and click on it to load it in your Code Editor. When you run the script, you will see digital elevation data from the National Aeronautics and Space Administration Shuttle Radar Topography Mission satellite visualized using two palettes, colorbrewer.Blues and cmocean.Algae. The map will have two layers that show the same data with different palettes. -The script first imports the digital elevation model data in the Imports section of the Code Editor. +The script first imports the digital elevation model data in the Imports section of the Code Editor. -var dem = ee.Image('USGS/SRTMGL1_003'); +var dem = ee.Image('USGS/SRTMGL1_003'); -The script then loads the ee-palettes module by using the require function. The path to the module, 'users/gena/packages:palettes', is passed to the function. The require function is then stored in a variable named 'palettes', which will be used later to obtain the palettes for data visualization. +The script then loads the ee-palettes module by using the require function. The path to the module, 'users/gena/packages:palettes', is passed to the function. The require function is then stored in a variable named 'palettes', which will be used later to obtain the palettes for data visualization. -var palettes = require('users/gena/packages:palettes'); +var palettes = require('users/gena/packages:palettes'); -As described by Donchyts et al. (2019), “Each palette is defined by a group and a name, which are separated by a period (JS object dot notation), and a color level. To retrieve a desired palette, use JS object notation to specify the group, name, and number of color levels.” We define the color palette Algae as palettes.cmocean.Algae[7] because it is part of the group cmocean and has 7 color levels. In the next code block, you can see that the palettes (i.e., lists of hex colors) have been defined for use by setting them as the value for the palette key in the visParams object supplied to the Map.addLayer function. +As described by Donchyts et al. (2019), “Each palette is defined by a group and a name, which are separated by a period (JS object dot notation), and a color level. To retrieve a desired palette, use JS object notation to specify the group, name, and number of color levels.” We define the color palette Algae as palettes.cmocean.Algae[7] because it is part of the group cmocean and has 7 color levels. In the next code block, you can see that the palettes (i.e., lists of hex colors) have been defined for use by setting them as the value for the palette key in the visParams object supplied to the Map.addLayer function. +```js // colorbrewer Map.addLayer(dem, { -   min: 0, -   max: 3000, -   palette: palettes.colorbrewer.Blues[9] + min: 0, + max: 3000, + palette: palettes.colorbrewer.Blues[9] }, 'colorbrewer Blues[9]'); // cmocean Map.addLayer(dem, { -   min: 0, -   max: 3000, -   palette: palettes.cmocean.Algae[7] + min: 0, + max: 3000, + palette: palettes.cmocean.Algae[7] }, 'cmocean Algae[7]'); -Question 8. Try adding a third layer to the Map with a different color palette from ee-palettes. How easy was it to do? +``` +Question 8. Try adding a third layer to the Map with a different color palette from ee-palettes. How easy was it to do? Now that you have loaded and used a module shared by someone else, you can try your hand at creating your own module and sharing it with someone else in your group. First, go to the shared repository that you created in Sect. 3, create a new script in that repository, and name it “cloudmasking.” -Then, go to the Examples repository at the bottom of the Scripts tab and select a function from the Cloud Masking repository. Let’s use the Landsat8 Surface Reflectance cloud masking script as an example. In that script, you will see the code shown in the block below. Copy all of it into your empty script.   +Then, go to the Examples repository at the bottom of the Scripts tab and select a function from the Cloud Masking repository. Let’s use the Landsat8 Surface Reflectance cloud masking script as an example. In that script, you will see the code shown in the block below. Copy all of it into your empty script. +```js // This example demonstrates the use of the Landsat 8 Collection 2, Level 2 // QA_PIXEL band (CFMask) to mask unwanted pixels. -function maskL8sr(image) {   // Bit 0 - Fill   // Bit 1 - Dilated Cloud   // Bit 2 - Cirrus   // Bit 3 - Cloud   // Bit 4 - Cloud Shadow   var qaMask = image.select('QA_PIXEL').bitwiseAnd(parseInt('11111',       2)).eq(0);   var saturationMask = image.select('QA_RADSAT').eq(0);   // Apply the scaling factors to the appropriate bands.   var opticalBands = image.select('SR_B.').multiply(0.0000275).add(-       0.2);   var thermalBands = image.select('ST_B.*').multiply(0.00341802) -       .add(149.0);   // Replace the original bands with the scaled ones and apply the masks.   return image.addBands(opticalBands, null, true) -       .addBands(thermalBands, null, true) -       .updateMask(qaMask) -       .updateMask(saturationMask); +function maskL8sr(image) { // Bit 0 - Fill // Bit 1 - Dilated Cloud // Bit 2 - Cirrus // Bit 3 - Cloud // Bit 4 - Cloud Shadow var qaMask = image.select('QA_PIXEL').bitwiseAnd(parseInt('11111', 2)).eq(0); var saturationMask = image.select('QA_RADSAT').eq(0); // Apply the scaling factors to the appropriate bands. var opticalBands = image.select('SR_B.').multiply(0.0000275).add(- 0.2); var thermalBands = image.select('ST_B.*').multiply(0.00341802) + .add(149.0); // Replace the original bands with the scaled ones and apply the masks. return image.addBands(opticalBands, null, true) + .addBands(thermalBands, null, true) + .updateMask(qaMask) + .updateMask(saturationMask); } // Map the function over one year of data. -var collection = ee.ImageCollection('LANDSAT/LC08/C02/T1_L2') -   .filterDate('2020-01-01', '2021-01-01') -   .map(maskL8sr); +var collection = ee.ImageCollection('LANDSAT/LC08/C02/T1_L2') + .filterDate('2020-01-01', '2021-01-01') + .map(maskL8sr); -var composite = collection.median(); +var composite = collection.median(); // Display the results. Map.setCenter(-4.52, 40.29, 7); // Iberian Peninsula Map.addLayer(composite, { -   bands: ['SR_B4', 'SR_B3', 'SR_B2'], -   min: 0, -   max: 0.3 + bands: ['SR_B4', 'SR_B3', 'SR_B2'], + min: 0, + max: 0.3 }); +``` Note that this code is well commented and has a header that describes what the script does. Don’t forget to comment your code and describe what you are doing each step of the way. This is a good practice for collaborative coding and for your own future reference. -Imagine that you changed this maskL8sr function slightly for some reason and want to make it available to other users and scripts. To do that, you can turn the function into a module. Copy and modify the code from the example code into the new script you created called “cloudmasking.” (Hint: Store the function in a variable starting with exports. Be careful that you don’t accidentally use Export, which is used to export datasets.) +Imagine that you changed this maskL8sr function slightly for some reason and want to make it available to other users and scripts. To do that, you can turn the function into a module. Copy and modify the code from the example code into the new script you created called “cloudmasking.” (Hint: Store the function in a variable starting with exports. Be careful that you don’t accidentally use Export, which is used to export datasets.) Your script should be similar to the following code. -exports.maskL8sr = function(image) {   // Bit 0 - Fill   // Bit 1 - Dilated Cloud   // Bit 2 - Cirrus   // Bit 3 - Cloud   // Bit 4 - Cloud Shadow   var qaMask = image.select('QA_PIXEL').bitwiseAnd(parseInt(       '11111', 2)).eq(0);   var saturationMask = image.select('QA_RADSAT').eq(0);   // Apply the scaling factors to the appropriate bands.   var opticalBands = image.select('SR_B.').multiply(0.0000275) -       .add(-0.2);   var thermalBands = image.select('ST_B.*').multiply(0.00341802) -       .add(149.0);   // Replace the original bands with the scaled ones and apply the masks.   return image.addBands(opticalBands, null, true) -       .addBands(thermalBands, null, true) -       .updateMask(qaMask) -       .updateMask(saturationMask); +exports.maskL8sr = function(image) { // Bit 0 - Fill // Bit 1 - Dilated Cloud // Bit 2 - Cirrus // Bit 3 - Cloud // Bit 4 - Cloud Shadow var qaMask = image.select('QA_PIXEL').bitwiseAnd(parseInt( '11111', 2)).eq(0); var saturationMask = image.select('QA_RADSAT').eq(0); // Apply the scaling factors to the appropriate bands. var opticalBands = image.select('SR_B.').multiply(0.0000275) + .add(-0.2); var thermalBands = image.select('ST_B.*').multiply(0.00341802) + .add(149.0); // Replace the original bands with the scaled ones and apply the masks. return image.addBands(opticalBands, null, true) + .addBands(thermalBands, null, true) + .updateMask(qaMask) + .updateMask(saturationMask); } Next, you will create a test script that makes use of the cloud masking module you just made. Begin by creating a new script in your shared repository called “cloudmasking-test.” You can modify the last part of the example cloud masking script to use your module. +```js // Map the function over one year of data. -var collection = ee.ImageCollection('LANDSAT/LC08/C02/T1_L2') -   .filterDate('2020-01-01', '2021-01-01') -   .map(maskL8sr); +var collection = ee.ImageCollection('LANDSAT/LC08/C02/T1_L2') + .filterDate('2020-01-01', '2021-01-01') + .map(maskL8sr); -var composite = collection.median(); +var composite = collection.median(); // Display the results. Map.setCenter(-4.52, 40.29, 7); // Iberian Peninsula Map.addLayer(composite, { -   bands: ['SR_B4', 'SR_B3', 'SR_B2'], -   min: 0, -   max: 0.3 + bands: ['SR_B4', 'SR_B3', 'SR_B2'], + min: 0, + max: 0.3 }); -Question 9. How will you modify the cloud masking script to use your module? What does the script look like? +``` +Question 9. How will you modify the cloud masking script to use your module? What does the script look like? -Answer: Your code might look something like the code block below. +Answer: Your code might look something like the code block below. +```js // Load the module -var myCloudFunctions = require(   'users/myusername/my-shared-repo:cloudmasking'); +var myCloudFunctions = require( 'users/myusername/my-shared-repo:cloudmasking'); // Map the function over one year of data. -var collection = ee.ImageCollection('LANDSAT/LC08/C02/T1_L2') -   .filterDate('2020-01-01', '2021-01-01') -   .map(myCloudFunctions.maskL8sr); +var collection = ee.ImageCollection('LANDSAT/LC08/C02/T1_L2') + .filterDate('2020-01-01', '2021-01-01') + .map(myCloudFunctions.maskL8sr); -var composite = collection.median(); +var composite = collection.median(); // Display the results. Map.setCenter(-4.52, 40.29, 7); // Iberian Peninsula Map.addLayer(composite, { -   bands: ['SR_B4', 'SR_B3', 'SR_B2'], -   min: 0, -   max: 0.3 + bands: ['SR_B4', 'SR_B3', 'SR_B2'], + min: 0, + max: 0.3 }); +``` ## Synthesis {.unnumbered} Apply what you learned in this chapter by setting up a shared repository for your project, lab group, or organization. What scripts would you share? What permissions should different users have? Are there any scripts that you would turn into modules? @@ -1099,14 +1115,14 @@ Donchyts G, Baart F, Braaten J (2019) ee-palettes. https://github.com/gee-commun -::: {.callout-tip} +:::{.callout-tip} # Chapter Information ## Author {.unlisted .unnumbered} -Jillian M. Deines, Stefania Di Tommaso, Nicholas Clinton, Noel Gorelick     +Jillian M. Deines, Stefania Di Tommaso, Nicholas Clinton, Noel Gorelick @@ -1129,10 +1145,10 @@ Commonly, when Earth Engine users move from tutorials to developing their own pr * Import images and image collections, filter, and visualize (Part F1). * Write a function and map it over an ImageCollection (Chap. F4.0). * Export and import results as Earth Engine assets (Chap. F5.0). -* Understand distinctions among Image, ImageCollection, Feature and FeatureCollection Earth Engine objects (Part F1, Part F2, Part F5). -* Use the require function to load code from existing modules (Chap. F6.1). +* Understand distinctions among Image, ImageCollection, Feature and FeatureCollection Earth Engine objects (Part F1, Part F2, Part F5). +* Use the require function to load code from existing modules (Chap. F6.1). -::: +::: ## Introduction {.unlisted .unnumbered} @@ -1151,13 +1167,13 @@ As you use Earth Engine, you may begin to have questions about how it works and Earth Engine is a parallel, distributed system (see Gorelick et al. 2017), which means that when you submit tasks, it breaks up pieces of your query onto different processors to complete them more efficiently. It then collects the results and returns them to you. For many users, not having to manually design this parallel, distributed processing is a huge benefit. For some advanced users, it can be frustrating to not have better control. We’d argue that leaving the details up to Earth Engine is a huge time-saver for most cases, and learning to work within a few constraints is a good time investment. -One core concept useful to master is the relationship between client-side and server-side operations. Client-side operations are performed within your browser (for the JavaScript API Code Editor) or local system (for the Python API). These include things such as manipulating strings or numbers in JavaScript. Server-side operations are executed on Google’s servers and include all of the ee.* functions. By using the Earth Engine APIs—JavaScript or Python—you are building a chain of commands to send to the servers and later receive the result back. As much as possible, you want to structure your code to send all the heavy lifting to Google, and keep processing off of your local resources. +One core concept useful to master is the relationship between client-side and server-side operations. Client-side operations are performed within your browser (for the JavaScript API Code Editor) or local system (for the Python API). These include things such as manipulating strings or numbers in JavaScript. Server-side operations are executed on Google’s servers and include all of the ee.* functions. By using the Earth Engine APIs—JavaScript or Python—you are building a chain of commands to send to the servers and later receive the result back. As much as possible, you want to structure your code to send all the heavy lifting to Google, and keep processing off of your local resources. -In other words, your work in the Code Editor is making a description of a computation. All ee objects are just placeholders for server-side objects—their actual value does not exist locally on your computer. To see or use the actual value, it has to be evaluated by the server. If you print an Earth Engine object, it calls getInfo to evaluate and return the value. In contrast, you can also work with JavaScript/Python lists or numbers locally, and do basic JavaScript/Python things to them, like add numbers together or loop over items. These are client-side objects. Whenever you bring a server-side object into your local environment, there’s a computational cost. +In other words, your work in the Code Editor is making a description of a computation. All ee objects are just placeholders for server-side objects—their actual value does not exist locally on your computer. To see or use the actual value, it has to be evaluated by the server. If you print an Earth Engine object, it calls getInfo to evaluate and return the value. In contrast, you can also work with JavaScript/Python lists or numbers locally, and do basic JavaScript/Python things to them, like add numbers together or loop over items. These are client-side objects. Whenever you bring a server-side object into your local environment, there’s a computational cost. Table F6.2.1 describes some nuts and bolts about Earth Engine and their implications. Table F6.2.2 provides some of the existing limits on individual tasks. -Table F6.2.1 Characterics of Google Earth Engine and implications for running large jobs  +Table F6.2.1 Characterics of Google Earth Engine and implications for running large jobs Earth Engine characteristics @@ -1169,7 +1185,7 @@ Occasionally, doing the exact same thing in two different orders can result in d Most processing is done per tile (generally a square that is 256 x 256 pixels). -Tasks that require many tiles are the most memory intensive. Some functions have a tileScale argument that reduces tile size, allowing processing-intensive jobs to succeed (at the cost of reduced speed). +Tasks that require many tiles are the most memory intensive. Some functions have a tileScale argument that reduces tile size, allowing processing-intensive jobs to succeed (at the cost of reduced speed). Export mode has higher memory and time allocations than interactive mode. @@ -1187,7 +1203,7 @@ The image processing domain, scale, and projection are defined by the specified There are not many cases when you will need to manually reproject images, and these operations are costly. Similarly, manually “clipping” images is typically unnecessary. -Table F6.2.2 Size limits for Earth Engine tasks +Table F6.2.2 Size limits for Earth Engine tasks Earth Engine Component @@ -1199,7 +1215,7 @@ Can print up to 5000 records. Computations must finish within five minutes. Export mode -Jobs have no time limit as long as they continue to make reasonable progress (defined roughly as 600 seconds per feature, two hours per aggregation, and 10 minutes per tile). If any one tile, feature, or aggregation takes too long, the whole job will get canceled. Any jobs that take longer than one week to run will likely fail due to Earth Engine's software update release cycles. +Jobs have no time limit as long as they continue to make reasonable progress (defined roughly as 600 seconds per feature, two hours per aggregation, and 10 minutes per tile). If any one tile, feature, or aggregation takes too long, the whole job will get canceled. Any jobs that take longer than one week to run will likely fail due to Earth Engine's software update release cycles. Table assets @@ -1209,9 +1225,9 @@ Maximum of 100 million features, 1000 properties (columns), and 100,000 vertices Good code scales better than bad code. But what is good code? Generally, for Earth Engine, good code means 1) using Earth Engine’s server-side operators; 2) avoiding multiple passes through the same image collection; 3) avoiding unnecessary conversions; and 4) setting the processing scale or sample numbers appropriate for your use case, i.e., avoid using very fine scales or large samples without reason. -We encourage readers to become familiar with the “Coding Best Practices” page in the online Earth Engine User Guide. This page provides examples for avoiding mixing client- and server-side functions, unnecessary conversions, costly algorithms, combining reducers, and other helpful tips. Similarly, the “Debugging Guide–Scaling Errors” page of the online Earth Engine User Guide covers some common problems and solutions. +We encourage readers to become familiar with the “Coding Best Practices” page in the online Earth Engine User Guide. This page provides examples for avoiding mixing client- and server-side functions, unnecessary conversions, costly algorithms, combining reducers, and other helpful tips. Similarly, the “Debugging Guide–Scaling Errors” page of the online Earth Engine User Guide covers some common problems and solutions. -In addition, some Earth Engine functions are more efficient than others. For example, Image.reduceRegions is more efficient than Image.sampleRegions, because sampleRegions regenerates the geometries under the hood. These types of best practices are trickier to enumerate and somewhat idiosyncratic. We encourage users to learn about and make use of the Profiler tab, which will track and display the resources used for each operation within your script. This can help identify areas to focus efficiency improvements. Note that the profiler itself increases resource use, so only use it when necessary to develop a script and remove it for production-level execution. Other ways to discover best practices include following/posting questions to GIS StackExchange or the Earth Engine Developer’s Discussion Group, swapping code with others, and experimentation. +In addition, some Earth Engine functions are more efficient than others. For example, Image.reduceRegions is more efficient than Image.sampleRegions, because sampleRegions regenerates the geometries under the hood. These types of best practices are trickier to enumerate and somewhat idiosyncratic. We encourage users to learn about and make use of the Profiler tab, which will track and display the resources used for each operation within your script. This can help identify areas to focus efficiency improvements. Note that the profiler itself increases resource use, so only use it when necessary to develop a script and remove it for production-level execution. Other ways to discover best practices include following/posting questions to GIS StackExchange or the Earth Engine Developer’s Discussion Group, swapping code with others, and experimentation. ## Scaling Across Time @@ -1222,26 +1238,26 @@ In this section we use an example of extracting climate data at features (points Earth Engine’s operators are designed to parallelize queries on the backend without user intervention. In many cases, they are sufficient to accomplish a scaling operation. -As an example, we will extract a daily time series of precipitation, maximum temperature, and minimum temperature for county polygons in the United States. We will use the GRIDMET Climate Reanalysis product (Abatzoglou 2013), which provides daily, 4000 m resolution gridded meteorological data from 1979 to the present across the contiguous United States. To save time for this practicum, we will focus on the states of Indiana, Illinois, and Iowa in the central United States, which together include 293 counties (Fig. F6.2.1). +As an example, we will extract a daily time series of precipitation, maximum temperature, and minimum temperature for county polygons in the United States. We will use the GRIDMET Climate Reanalysis product (Abatzoglou 2013), which provides daily, 4000 m resolution gridded meteorological data from 1979 to the present across the contiguous United States. To save time for this practicum, we will focus on the states of Indiana, Illinois, and Iowa in the central United States, which together include 293 counties (Fig. F6.2.1). -![](F6/image11.png) +![Fig. F6.2.1 Map of study area, showing 293 county features within the states of Iowa, Illinois, and Indiana in the United States](F6/image11.png) -Fig. F6.2.1 Map of study area, showing 293 county features within the states of Iowa, Illinois, and Indiana in the United States -This example uses the ee.Image.reduceRegions operator, which extracts statistics from an Image for each Feature (point or polygon) in a FeatureCollection. We will map the reduceRegions operator over each daily image in an ImageCollection, thus providing us with the daily climate information for each county of interest. +This example uses the ee.Image.reduceRegions operator, which extracts statistics from an Image for each Feature (point or polygon) in a FeatureCollection. We will map the reduceRegions operator over each daily image in an ImageCollection, thus providing us with the daily climate information for each county of interest. Note that although our example uses a climate ImageCollection, this approach transfers to any ImageCollection, including satellite imagery, as well as image collections that you have already processed, such as cloud masking (Chap. F4.3) or time series aggregation (Chap. F4.2). First, define the FeatureCollection, ImageCollection, and time period: +```js // Load county dataset. // Filter counties in Indiana, Illinois, and Iowa by state FIPS code. // Select only the unique ID column for simplicity. -var countiesAll = ee.FeatureCollection('TIGER/2018/Counties'); -var states = ['17', '18', '19']; -var uniqueID = 'GEOID'; -var featColl = countiesAll.filter(ee.Filter.inList('STATEFP', states)) -   .select(uniqueID); +var countiesAll = ee.FeatureCollection('TIGER/2018/Counties'); +var states = ['17', '18', '19']; +var uniqueID = 'GEOID'; +var featColl = countiesAll.filter(ee.Filter.inList('STATEFP', states)) + .select(uniqueID); print(featColl.size()); print(featColl.limit(1)); @@ -1251,98 +1267,105 @@ Map.centerObject(featColl, 5); Map.addLayer(featColl); // specify years of interest -var startYear = 2020; -var endYear = 2020; +var startYear = 2020; +var endYear = 2020; // climate dataset info -var imageCollectionName = 'IDAHO_EPSCOR/GRIDMET'; -var bandsWanted = ['pr', 'tmmn', 'tmmx']; -var scale = 4000; +var imageCollectionName = 'IDAHO_EPSCOR/GRIDMET'; +var bandsWanted = ['pr', 'tmmn', 'tmmx']; +var scale = 4000; -Printing the size of the FeatureCollection indicates that there are 293 counties in our subset. Since we want to pull a daily time series for one year, our final dataset will have 106,945 rows—one for each county-day. +``` +Printing the size of the FeatureCollection indicates that there are 293 counties in our subset. Since we want to pull a daily time series for one year, our final dataset will have 106,945 rows—one for each county-day. -Note that from our county FeatureCollection, we select only the GEOID column, which represents a unique identifier for each record in this dataset. We do this here to simplify print outputs; we could also specify which properties to include in the export function (see below). +Note that from our county FeatureCollection, we select only the GEOID column, which represents a unique identifier for each record in this dataset. We do this here to simplify print outputs; we could also specify which properties to include in the export function (see below). -Next, load and filter the climate data. Note we adjust the end date to January 1 of the following year, rather than December 31 of the specified year, since the filterDate function has an inclusive start date argument and an exclusive end date argument; without this modification the output would lack data for December 31. +Next, load and filter the climate data. Note we adjust the end date to January 1 of the following year, rather than December 31 of the specified year, since the filterDate function has an inclusive start date argument and an exclusive end date argument; without this modification the output would lack data for December 31. +```js // Load and format climate data. -var startDate = startYear + '-01-01'; +var startDate = startYear + '-01-01'; -var endYear_adj = endYear + 1; -var endDate = endYear_adj + '-01-01'; +var endYear_adj = endYear + 1; +var endDate = endYear_adj + '-01-01'; -var imageCollection = ee.ImageCollection(imageCollectionName) -   .select(bandsWanted) -   .filterBounds(featColl) -   .filterDate(startDate, endDate); +var imageCollection = ee.ImageCollection(imageCollectionName) + .select(bandsWanted) + .filterBounds(featColl) + .filterDate(startDate, endDate); -Now get the mean value for each climate attribute within each county feature. Here, we map the ee.Image.reduceRegions call over the ImageCollection, specifying an ee.Reducer.mean reducer. The reducer will apply to each band in the image, and it returns the FeatureCollection with new properties. We also add a 'date_ymd' time property extracted from the image to correctly associate daily values with their date. Finally, we flatten the output to reform a single FeatureCollection with one feature per county-day. +``` +Now get the mean value for each climate attribute within each county feature. Here, we map the ee.Image.reduceRegions call over the ImageCollection, specifying an ee.Reducer.mean reducer. The reducer will apply to each band in the image, and it returns the FeatureCollection with new properties. We also add a 'date_ymd' time property extracted from the image to correctly associate daily values with their date. Finally, we flatten the output to reform a single FeatureCollection with one feature per county-day. +```js // get values at features -var sampledFeatures = imageCollection.map(function(image) {   return image.reduceRegions({ -           collection: featColl, -           reducer: ee.Reducer.mean(), -           scale: scale -       }).filter(ee.Filter.notNull( -       bandsWanted)) // drop rows with no data       .map(function(f) { // add date property           var time_start = image.get(               'system:time_start');           var dte = ee.Date(time_start).format(               'YYYYMMdd');           return f.set('date_ymd', dte); -       }); +var sampledFeatures = imageCollection.map(function(image) { return image.reduceRegions({ + collection: featColl, + reducer: ee.Reducer.mean(), + scale: scale + }).filter(ee.Filter.notNull( + bandsWanted)) // drop rows with no data .map(function(f) { // add date property var time_start = image.get( 'system:time_start'); var dte = ee.Date(time_start).format( 'YYYYMMdd'); return f.set('date_ymd', dte); + }); }).flatten(); print(sampledFeatures.limit(1)); -Note that we include a filter to remove feature-day rows that lacked data. While this is less common when using gridded climate products, missing data can be common when reducing satellite images. This is because satellite collections come in scene tiles, and each image tile likely does not overlap all of our features unless it has first been aggregated temporally. It can also occur if a cloud mask has been applied to an image prior to the reduction. By filtering out null values, we can reduce empty rows. +``` +Note that we include a filter to remove feature-day rows that lacked data. While this is less common when using gridded climate products, missing data can be common when reducing satellite images. This is because satellite collections come in scene tiles, and each image tile likely does not overlap all of our features unless it has first been aggregated temporally. It can also occur if a cloud mask has been applied to an image prior to the reduction. By filtering out null values, we can reduce empty rows. -Now explore the result. If we simply print(sampledFeatures) we get our first error message: “User memory limit exceeded.” This is because we’ve created a FeatureCollection that exceeds the size limits set for interactive mode. How many are there? We could try print(sampledFeatures.size()), but due to the larger size, we receive a “Computation timed out” message—it’s unable to tell us. Of course, we know that we expect 293 counties x 365 days = 106,945 features. We can, however, check that our reducer has worked as expected by asking Earth Engine for just one feature: print(sampledFeatures.limit(1)). +Now explore the result. If we simply print(sampledFeatures) we get our first error message: “User memory limit exceeded.” This is because we’ve created a FeatureCollection that exceeds the size limits set for interactive mode. How many are there? We could try print(sampledFeatures.size()), but due to the larger size, we receive a “Computation timed out” message—it’s unable to tell us. Of course, we know that we expect 293 counties x 365 days = 106,945 features. We can, however, check that our reducer has worked as expected by asking Earth Engine for just one feature: print(sampledFeatures.limit(1)). -![](F6/image75.png) +![Fig. F6.2.2 Screenshot of the print output for one feature after the reduceRegions call](F6/image75.png) -Fig. F6.2.2 Screenshot of the print output for one feature after the reduceRegions call Here, we can see the precipitation, minimum temperature, and maximum temperature for the county with GEOID = 17121 on January 1, 2020 (Fig. F6.2.2; note temperature is in Kelvin units). -Next, export the full FeatureCollection as a CSV to a folder in your Google Drive. Specify the names of properties to include. Build part of the filename dynamically based on arguments used for year and data scale, so we don’t need to manually modify the filenames. +Next, export the full FeatureCollection as a CSV to a folder in your Google Drive. Specify the names of properties to include. Build part of the filename dynamically based on arguments used for year and data scale, so we don’t need to manually modify the filenames. +```js // export info -var exportFolder = 'GEE_scalingUp'; -var filename = 'Gridmet_counties_IN_IL_IA_' + scale + 'm_' + -   startYear + '-' + endYear;// prepare export: specify properties/columns to include -var columnsWanted = [uniqueID].concat(['date_ymd'], bandsWanted); +var exportFolder = 'GEE_scalingUp'; +var filename = 'Gridmet_counties_IN_IL_IA_' + scale + 'm_' + + startYear + '-' + endYear;// prepare export: specify properties/columns to include +var columnsWanted = [uniqueID].concat(['date_ymd'], bandsWanted); print(columnsWanted); Export.table.toDrive({ -   collection: sampledFeatures, -   description: filename, -   folder: exportFolder, -   fileFormat: 'CSV', -   selectors: columnsWanted + collection: sampledFeatures, + description: filename, + folder: exportFolder, + fileFormat: 'CSV', + selectors: columnsWanted }); -::: {.callout-note} -Code Checkpoint F62a. The book’s repository contains a script that shows what your code should look like at this point. +``` +:::{.callout-note} +Code Checkpoint F62a. The book’s repository contains a script that shows what your code should look like at this point. ::: On our first export, this job took about eight minutes to complete, producing a dataset 6.8 MB in size. The data is ready for downstream use but may need formatting to suit the user’s goals. You can see what the exported CSV looks like in Fig. F6.2.3. -![](F6/image41.png) +![Fig. F6.2.3 Top six rows of the exported CSV viewed in Microsoft Excel and sorted by county GEOID ](F6/image41.png) -Fig. F6.2.3 Top six rows of the exported CSV viewed in Microsoft Excel and sorted by county GEOID  Using the Selectors Argument -There are two excellent reasons to use the selectors argument in your Export.table.toDrive call. First, if the argument is not specified, Earth Engine will generate the column names for the exported CSV from the first feature in your FeatureCollection. If that feature is missing properties, those properties will be dropped from the export for all features. +There are two excellent reasons to use the selectors argument in your Export.table.toDrive call. First, if the argument is not specified, Earth Engine will generate the column names for the exported CSV from the first feature in your FeatureCollection. If that feature is missing properties, those properties will be dropped from the export for all features. -Perhaps even more important if you are seeking to scale up an analysis, including unnecessary columns can greatly increase file size and even processing time. For example, Earth Engine includes a .geo field that contains a GeoJSON description of each spatial feature. For non-simple geometries, the field can be quite large, as it lists coordinates for each polygon vertex. For many purposes, it’s not necessary to include this information for each daily record (here, 365 daily rows per feature). +Perhaps even more important if you are seeking to scale up an analysis, including unnecessary columns can greatly increase file size and even processing time. For example, Earth Engine includes a .geo field that contains a GeoJSON description of each spatial feature. For non-simple geometries, the field can be quite large, as it lists coordinates for each polygon vertex. For many purposes, it’s not necessary to include this information for each daily record (here, 365 daily rows per feature). -For example, when we ran the same job as above but did not use the selectors argument, the output dataset was 5.7 GB (versus 6.8 MB!) and the runtime was slower. This is a cumbersomely large file, with no real benefit. We generally recommend dropping the .geo column and other unnecessary properties. To retain spatial information, a unique identifier for each feature can be used for downstream joins with the spatial data or other properties. If working with point data, latitude and longitude columns can be added prior to export to maintain easily accessible geographic information, although the .geo column for point data is far smaller than for irregularly shaped polygon features. +For example, when we ran the same job as above but did not use the selectors argument, the output dataset was 5.7 GB (versus 6.8 MB!) and the runtime was slower. This is a cumbersomely large file, with no real benefit. We generally recommend dropping the .geo column and other unnecessary properties. To retain spatial information, a unique identifier for each feature can be used for downstream joins with the spatial data or other properties. If working with point data, latitude and longitude columns can be added prior to export to maintain easily accessible geographic information, although the .geo column for point data is far smaller than for irregularly shaped polygon features. ### 1.2. Scaling Across Time by Batching: Get 20 Years of Daily Climate Data Above, we extracted one year of daily data for our 293 counties. Let’s say we want to do the same thing, but for 2001–2020. We have already written our script to flexibly specify years, so it’s fairly adaptable to this new use case: +```js // specify years of interest -var startYear = 2020; -var endYear = 2020; +var startYear = 2020; +var endYear = 2020; -If we only wanted a few years for a small number of features, we could just modify the startYear or endYear and proceed. Indeed, our current example is modest in size and number of features, and we were able to run 2001–2020 in one export job that took about two hours, with an output file size of 299 MB. However, with larger feature collections, or hourly data, we will again start to bump up against Earth Engine’s limits. Generally, jobs of this sort do not fail quickly—exports are allowed to run as long as they continue making progress (see Table F6.2.2). It’s not uncommon, however, for a large job to take well over 24 hours to run, or even to fail after more than 24 hours of run time, as it accumulates too many records or a single aggregation fails. For users, this can be frustrating. +``` +If we only wanted a few years for a small number of features, we could just modify the startYear or endYear and proceed. Indeed, our current example is modest in size and number of features, and we were able to run 2001–2020 in one export job that took about two hours, with an output file size of 299 MB. However, with larger feature collections, or hourly data, we will again start to bump up against Earth Engine’s limits. Generally, jobs of this sort do not fail quickly—exports are allowed to run as long as they continue making progress (see Table F6.2.2). It’s not uncommon, however, for a large job to take well over 24 hours to run, or even to fail after more than 24 hours of run time, as it accumulates too many records or a single aggregation fails. For users, this can be frustrating. We generally find it simpler to run several small jobs rather than one large job. Outputs can then be combined in external software. This avoids any frustration with long-running jobs or delayed failures, and it allows parts of the task to be run simultaneously. Earth Engine generally executes from 2–20 jobs per user at a time, depending on overall user load (although 20 is rare). As a counterpart, there is some overhead for generating separate jobs. @@ -1352,79 +1375,81 @@ For-Loops: They Are Sometimes OK Batching jobs in time is a great way to break up a task into smaller units. Other options include batching jobs by spatial regions defined by polygons (see Sect. 2), or for computationally heavy tasks, batching by both space and time. -Because Export functions are client-side functions, however, you can’t create an export within an Earth Engine map command. Instead, we need to loop over the variable that will define our batches and create a set of export tasks. +Because Export functions are client-side functions, however, you can’t create an export within an Earth Engine map command. Instead, we need to loop over the variable that will define our batches and create a set of export tasks. But wait! Aren’t we supposed to avoid for-loops at all costs? Yes, within a computational chain. Here, we are using a loop to send multiple computational chains to the server. First, we will start with the same script as in Sect. 1.1, but we will modify the start year. We will also modify the desired output filename to be a generic base filename, to which we will append the year for each task within the loop (in the next step). +```js // Load county dataset. -var countiesAll = ee.FeatureCollection('TIGER/2018/Counties'); -var states = ['17', '18', '19']; -var uniqueID = 'GEOID'; -var featColl = countiesAll.filter(ee.Filter.inList('STATEFP', states)) -   .select(uniqueID); +var countiesAll = ee.FeatureCollection('TIGER/2018/Counties'); +var states = ['17', '18', '19']; +var uniqueID = 'GEOID'; +var featColl = countiesAll.filter(ee.Filter.inList('STATEFP', states)) + .select(uniqueID); print(featColl.size()); print(featColl.limit(1)); Map.addLayer(featColl); // Specify years of interest. -var startYear = 2001; -var endYear = 2020; +var startYear = 2001; +var endYear = 2020; // Climate dataset info. -var imageCollectionName = 'IDAHO_EPSCOR/GRIDMET'; -var bandsWanted = ['pr', 'tmmn', 'tmmx']; -var scale = 4000; +var imageCollectionName = 'IDAHO_EPSCOR/GRIDMET'; +var bandsWanted = ['pr', 'tmmn', 'tmmx']; +var scale = 4000; // Export info. -var exportFolder = 'GEE_scalingUp'; -var filenameBase = 'Gridmet_counties_IN_IL_IA_' + scale + 'm_'; +var exportFolder = 'GEE_scalingUp'; +var filenameBase = 'Gridmet_counties_IN_IL_IA_' + scale + 'm_'; -Now modify the code in Sect. 1.1 to use a looping variable, i, to represent each year. Here, we are using standard JavaScript looping syntax, where i will take on each value between our startYear (2001) and our endYear (2020) for each loop through this section of code, thus creating 20 queries to send to Earth Engine’s servers. +``` +Now modify the code in Sect. 1.1 to use a looping variable, i, to represent each year. Here, we are using standard JavaScript looping syntax, where i will take on each value between our startYear (2001) and our endYear (2020) for each loop through this section of code, thus creating 20 queries to send to Earth Engine’s servers. +```js // Initiate a loop, in which the variable i takes on values of each year. -for (var i = startYear; i <= endYear; i++) {        // for each year....  // Load climate collection for that year. var startDate = i + '-01-01'; +for (var i = startYear; i <= endYear; i++) { // for each year.... // Load climate collection for that year. var startDate = i + '-01-01'; -  var endYear_adj = i + 1; var endDate = endYear_adj + '-01-01'; var imageCollection = ee.ImageCollection(imageCollectionName) -     .select(bandsWanted) -     .filterBounds(featColl) -     .filterDate(startDate, endDate); // Get values at feature collection. var sampledFeatures = imageCollection.map(function(image) {   return image.reduceRegions({ -     collection: featColl, -     reducer: ee.Reducer.mean(),         -     tileScale: 1, -     scale: scale -   }).filter(ee.Filter.notNull(bandsWanted))  // remove rows without data     .map(function(f) {                  // add date property       var time_start = image.get('system:time_start');       var dte = ee.Date(time_start).format('YYYYMMdd');       return f.set('date_ymd', dte); -   }); - }).flatten(); // Prepare export: specify properties and filename. var columnsWanted = [uniqueID].concat(['date_ymd'], bandsWanted); var filename = filenameBase + i; Export.table.toDrive({ -   collection: sampledFeatures, -   description: filename, -   folder: exportFolder, -   fileFormat: 'CSV', -   selectors: columnsWanted - }); -  + var endYear_adj = i + 1; var endDate = endYear_adj + '-01-01'; var imageCollection = ee.ImageCollection(imageCollectionName) + .select(bandsWanted) + .filterBounds(featColl) + .filterDate(startDate, endDate); // Get values at feature collection. var sampledFeatures = imageCollection.map(function(image) { return image.reduceRegions({ + collection: featColl, + reducer: ee.Reducer.mean(), + tileScale: 1, + scale: scale + }).filter(ee.Filter.notNull(bandsWanted)) // remove rows without data .map(function(f) { // add date property var time_start = image.get('system:time_start'); var dte = ee.Date(time_start).format('YYYYMMdd'); return f.set('date_ymd', dte); + }); + }).flatten(); // Prepare export: specify properties and filename. var columnsWanted = [uniqueID].concat(['date_ymd'], bandsWanted); var filename = filenameBase + i; Export.table.toDrive({ + collection: sampledFeatures, + description: filename, + folder: exportFolder, + fileFormat: 'CSV', + selectors: columnsWanted + }); + } -::: {.callout-note} -Code Checkpoint F62b. The book’s repository contains a script that shows what your code should look like at this point. +:::{.callout-note} +Code Checkpoint F62b. The book’s repository contains a script that shows what your code should look like at this point. ::: -When we run this script, it builds our computational query for each year, creating a batch of 20 individual jobs that will show up in the Task pane (Fig. F6.2.4). Each task name includes the year, since we used our looping variable i to modify the base filename we specified. +When we run this script, it builds our computational query for each year, creating a batch of 20 individual jobs that will show up in the Task pane (Fig. F6.2.4). Each task name includes the year, since we used our looping variable i to modify the base filename we specified. -![](F6/image63.png) +![Fig. F6.2.4 Creation of batch tasks for each year](F6/image63.png) -Fig. F6.2.4 Creation of batch tasks for each year -We now encounter a downside to creating batch tasks within the JavaScript Code Editor: we need to click Run to execute each job in turn. Here, we made this easier by programmatically assigning each job the filename we want, so we can hold the Cmd/Ctrl key and click Run to avoid the export task option window and only need to click once per task. Still, one can imagine that at some number of tasks, one’s patience for clicking Run will be exceeded. We assume that number is different for everyone. +We now encounter a downside to creating batch tasks within the JavaScript Code Editor: we need to click Run to execute each job in turn. Here, we made this easier by programmatically assigning each job the filename we want, so we can hold the Cmd/Ctrl key and click Run to avoid the export task option window and only need to click once per task. Still, one can imagine that at some number of tasks, one’s patience for clicking Run will be exceeded. We assume that number is different for everyone. -Note: If at any time you have submitted several tasks to the server but want to cancel them all, you can do so more easily from the Earth Engine Task Manager that is linked at the top of the Task pane. You can read about that task manager in the Earth Engine User Guide. +Note: If at any time you have submitted several tasks to the server but want to cancel them all, you can do so more easily from the Earth Engine Task Manager that is linked at the top of the Task pane. You can read about that task manager in the Earth Engine User Guide. -In order to auto-execute jobs in batch mode, we’d need to use the Python API. Interested users can see the Earth Engine User Guide Python API tutorial for further details about the Python API.  +In order to auto-execute jobs in batch mode, we’d need to use the Python API. Interested users can see the Earth Engine User Guide Python API tutorial for further details about the Python API. ## Scaling Across Space via Spatial Tiling -Breaking up jobs in space is another key strategy for scaling operations in Earth Engine. Here, we will focus on making a cloud-free composite from the Sentinel-2 Level 2A Surface Reflectance product. The approach is similar to that in Chap. F4.3, which explores cloud-free compositing. The main difference is that Landsat scenes come with a reliable quality band for each scene, whereas the process for Sentinel-2 is a bit more complicated and computationally intense (see below). +Breaking up jobs in space is another key strategy for scaling operations in Earth Engine. Here, we will focus on making a cloud-free composite from the Sentinel-2 Level 2A Surface Reflectance product. The approach is similar to that in Chap. F4.3, which explores cloud-free compositing. The main difference is that Landsat scenes come with a reliable quality band for each scene, whereas the process for Sentinel-2 is a bit more complicated and computationally intense (see below). Our region of interest is the state of Washington in the United States for demonstration purposes, but the method will work at much larger continental scales as well. @@ -1435,87 +1460,93 @@ While we do not intend to cover the theory behind Sentinel-2 cloud masking, we d The Sentinel-2 Level 2A collection does not come with a robust cloud mask. Instead, we will build one from related products that have been developed for this purpose. Following the existing Sentinel-2 cloud masking tutorials in the Earth Engine guides, this approach requires three Sentinel-2 image collections: * The Sentinel-2 Level 2A Surface Reflectance product. This is the dataset we want to use to build our final composite. -* The Sentinel-2 Cloud Probability Dataset, an ImageCollection that contains cloud probabilities for each Sentinel-2 scene. -* The Sentinel-2 Level 1C top-of-atmosphere product. This collection is needed to run the Cloud Displacement Index to identify cloud shadows, which is calculated using ee.Algorithms.Sentinel2.CDI (see Frantz et al. 2018 for algorithm description). +* The Sentinel-2 Cloud Probability Dataset, an ImageCollection that contains cloud probabilities for each Sentinel-2 scene. +* The Sentinel-2 Level 1C top-of-atmosphere product. This collection is needed to run the Cloud Displacement Index to identify cloud shadows, which is calculated using ee.Algorithms.Sentinel2.CDI (see Frantz et al. 2018 for algorithm description). -These three image collections all contain 10 m resolution data for every Sentinel-2 scene. We will join them based on their 'system:index' property so we can relate each Level 2A scene with the corresponding cloud probability and cloud displacement index. Furthermore, there are two ee.Image.projection steps to control the scale when calculating clouds and their shadows. +These three image collections all contain 10 m resolution data for every Sentinel-2 scene. We will join them based on their 'system:index' property so we can relate each Level 2A scene with the corresponding cloud probability and cloud displacement index. Furthermore, there are two ee.Image.projection steps to control the scale when calculating clouds and their shadows. To sum up, the cloud masking approach is computationally costly, thus requiring some thought when applying it at scale. -### 2.1. Generate a Cloud-Free Satellite Composite: Limits to On-the-Fly Computing +### 2.1. Generate a Cloud-Free Satellite Composite: Limits to On-the-Fly Computing -Note: Our focus here is on code structure for implementing spatial tiling. Below, we import existing tested functions for cloud masking using the require command. +Note: Our focus here is on code structure for implementing spatial tiling. Below, we import existing tested functions for cloud masking using the require command. -First, define our region and time of interest; then, load the module containing the cloud functions. +First, define our region and time of interest; then, load the module containing the cloud functions. // Set the Region of Interest:Seattle, Washington, United States -var roi = ee.Geometry.Point([-122.33524518034544, 47.61356183942883]); +var roi = ee.Geometry.Point([-122.33524518034544, 47.61356183942883]); // Dates over which to create a median composite. -var start = ee.Date('2019-03-01'); -var end = ee.Date('2019-09-01'); +var start = ee.Date('2019-03-01'); +var end = ee.Date('2019-09-01'); // Specify module with cloud mask functions. -var s2mask_tools = require(   'projects/gee-edu/book:Part F - Fundamentals/F6 - Advanced Topics/F6.2 Scaling Up/modules/s2cloudmask.js' +var s2mask_tools = require( 'projects/gee-edu/book:Part F - Fundamentals/F6 - Advanced Topics/F6.2 Scaling Up/modules/s2cloudmask.js' ); +``` Next, load and filter our three Sentinel-2 image collections. +```js // Sentinel-2 surface reflectance data for the composite. -var s2Sr = ee.ImageCollection('COPERNICUS/S2_SR') -   .filterDate(start, end) -   .filterBounds(roi) -   .select(['B2', 'B3', 'B4', 'B5']); +var s2Sr = ee.ImageCollection('COPERNICUS/S2_SR') + .filterDate(start, end) + .filterBounds(roi) + .select(['B2', 'B3', 'B4', 'B5']); -// Sentinel-2 Level 1C data (top-of-atmosphere).   +// Sentinel-2 Level 1C data (top-of-atmosphere). // Bands B7, B8, B8A and B10 needed for CDI and the cloud mask function. -var s2 = ee.ImageCollection('COPERNICUS/S2') -   .filterBounds(roi) -   .filterDate(start, end) -   .select(['B7', 'B8', 'B8A', 'B10']); +var s2 = ee.ImageCollection('COPERNICUS/S2') + .filterBounds(roi) + .filterDate(start, end) + .select(['B7', 'B8', 'B8A', 'B10']); // Cloud probability dataset - used in cloud mask function -var s2c = ee.ImageCollection('COPERNICUS/S2_CLOUD_PROBABILITY') -   .filterDate(start, end) -   .filterBounds(roi); +var s2c = ee.ImageCollection('COPERNICUS/S2_CLOUD_PROBABILITY') + .filterDate(start, end) + .filterBounds(roi); +``` Now apply the cloud mask: +```js // Join the cloud probability dataset to surface reflectance. -var withCloudProbability = s2mask_tools.indexJoin(s2Sr, s2c,   'cloud_probability'); +var withCloudProbability = s2mask_tools.indexJoin(s2Sr, s2c, 'cloud_probability'); // Join the L1C data to get the bands needed for CDI. -var withS2L1C = s2mask_tools.indexJoin(withCloudProbability, s2,   'l1c'); +var withS2L1C = s2mask_tools.indexJoin(withCloudProbability, s2, 'l1c'); // Map the cloud masking function over the joined collection. // Cast output to ImageCollection -var masked = ee.ImageCollection(withS2L1C.map(s2mask_tools +var masked = ee.ImageCollection(withS2L1C.map(s2mask_tools .maskImage)); +``` Next, generate and visualize the median composite: +```js // Take the median, specifying a tileScale to avoid memory errors. -var median = masked.reduce(ee.Reducer.median(), 8); +var median = masked.reduce(ee.Reducer.median(), 8); // Display the results. Map.centerObject(roi, 12); Map.addLayer(roi); -var viz = { -   bands: ['B4_median', 'B3_median', 'B2_median'], -   min: 0, -   max: 3000 +var viz = { + bands: ['B4_median', 'B3_median', 'B2_median'], + min: 0, + max: 3000 }; Map.addLayer(median, viz, 'median'); -::: {.callout-note} -Code Checkpoint F62c. The book’s repository contains a script that shows what your code should look like at this point. +``` +:::{.callout-note} +Code Checkpoint F62c. The book’s repository contains a script that shows what your code should look like at this point. ::: After about 1–3 minutes, Earth Engine returns our composite to us on the fly (Fig. F6.2.5). Note that panning and zooming to a new area requires that Earth Engine must again issue the compositing request to calculate the image for new areas. Given the delay, this isn’t a very satisfying way to explore our composite. -![](F6/image49.png)![](F6/image43.png) +![Fig. F6.2.5 Map view of Seattle, Washington, USA (left) and the corresponding Sentinel-2 composite (right)](F6/image49.png)![Fig. F6.2.5 Map view of Seattle, Washington, USA (left) and the corresponding Sentinel-2 composite (right)](F6/image43.png) -Fig. F6.2.5 Map view of Seattle, Washington, USA (left) and the corresponding Sentinel-2 composite (right) Next, expand our view (set zoom to 9) to exceed the limits of on-the-fly computation (Fig. F6.2.6). @@ -1523,121 +1554,124 @@ Map.centerObject(roi, 9); Map.addLayer(roi); Map.addLayer(median, viz, 'median'); -![](F6/image42.png) +![Fig. F6.2.6 Error message for exceeding memory limits in interactive mode](F6/image42.png) -Fig. F6.2.6 Error message for exceeding memory limits in interactive mode As you can see, this is an excellent candidate for an export task rather than running in “on-the-fly” interactive mode, as above. -### 2.2. Generate a Regional Composite Through Spatial Tiling +### 2.2. Generate a Regional Composite Through Spatial Tiling -Our goal is to apply the cloud masking method in Sect. 2.1 to the state of Washington, United States. In our testing, we successfully exported one Sentinel-2 composite for this area in about nine hours, but for this tutorial, let’s presume we need to split the task up to be successful. +Our goal is to apply the cloud masking method in Sect. 2.1 to the state of Washington, United States. In our testing, we successfully exported one Sentinel-2 composite for this area in about nine hours, but for this tutorial, let’s presume we need to split the task up to be successful. -Essentially, we want to split our region of interest up into a regular grid. For each grid, we will export a composite image into a new ImageCollection asset. We can then load and mosaic our composite for use in downstream scripts (see below). +Essentially, we want to split our region of interest up into a regular grid. For each grid, we will export a composite image into a new ImageCollection asset. We can then load and mosaic our composite for use in downstream scripts (see below). First, generate a spatial polygon grid (FeatureCollection) of desired size over your region of interest (see Fig. F6.2.7): +```js // Specify helper functions. -var s2mask_tools = require(   'projects/gee-edu/book:Part F - Fundamentals/F6 - Advanced Topics/F6.2 Scaling Up/modules/s2cloudmask.js' +var s2mask_tools = require( 'projects/gee-edu/book:Part F - Fundamentals/F6 - Advanced Topics/F6.2 Scaling Up/modules/s2cloudmask.js' ); // Set the Region of Interest: Washington, USA. -var roi = ee.FeatureCollection('TIGER/2018/States') -   .filter(ee.Filter.equals('NAME', 'Washington')); +var roi = ee.FeatureCollection('TIGER/2018/States') + .filter(ee.Filter.equals('NAME', 'Washington')); // Specify grid size in projection, x and y units (based on projection). -var projection = 'EPSG:4326'; // WGS84 lat lon -var dx = 2.5; -var dy = 1.5; +var projection = 'EPSG:4326'; // WGS84 lat lon +var dx = 2.5; +var dy = 1.5; // Dates over which to create a median composite. -var start = ee.Date('2019-03-01'); -var end = ee.Date('2019-09-01'); +var start = ee.Date('2019-03-01'); +var end = ee.Date('2019-09-01'); // Make grid and visualize. -var proj = ee.Projection(projection).scale(dx, dy); -var grid = roi.geometry().coveringGrid(proj); +var proj = ee.Projection(projection).scale(dx, dy); +var grid = roi.geometry().coveringGrid(proj); Map.addLayer(roi, {}, 'roi'); Map.addLayer(grid, {}, 'grid'); -![](F6/image12.png) +``` +![Fig. F6.2.7 Visualization of the regular spatial grid generated for use in spatial batch processing](F6/image12.png) -Fig. F6.2.7 Visualization of the regular spatial grid generated for use in spatial batch processing -Next, create a new, empty ImageCollection asset to use as our export destination (Assets > New > Image Collection; Fig. F6.2.8). Name the image collection 'S2_composite_WA' and specify the asset location in your user folder (e.g., "path/to/your/asset/s2_composite_WA"). +Next, create a new, empty ImageCollection asset to use as our export destination (Assets > New > Image Collection; Fig. F6.2.8). Name the image collection 'S2_composite_WA' and specify the asset location in your user folder (e.g., "path/to/your/asset/s2_composite_WA"). -![](F6/image15.png) +![Fig. F6.2.8 The “create new image collection asset” menu in the Code Editor](F6/image15.png) -Fig. F6.2.8 The “create new image collection asset” menu in the Code Editor -Specify the ImageCollection to export to, along with a base name for each image (the tile number will be appended in the batch export). +Specify the ImageCollection to export to, along with a base name for each image (the tile number will be appended in the batch export). +```js // Export info. -var assetCollection = 'path/to/your/asset/s2_composite_WA'; -var imageBaseName = 'S2_median_'; +var assetCollection = 'path/to/your/asset/s2_composite_WA'; +var imageBaseName = 'S2_median_'; -Extract grid numbers to use as looping variables. Note there is one getInfo call here, which should be used sparingly and never within a for-loop if you can help it. We use it to bring the number of grid cells we’ve generated onto the client-side to set up the for-loop over grids. Note that if your grid has too many elements, you may need a different strategy. +``` +Extract grid numbers to use as looping variables. Note there is one getInfo call here, which should be used sparingly and never within a for-loop if you can help it. We use it to bring the number of grid cells we’ve generated onto the client-side to set up the for-loop over grids. Note that if your grid has too many elements, you may need a different strategy. +```js // Get a list based on grid number. -var gridSize = grid.size().getInfo(); -var gridList = grid.toList(gridSize); +var gridSize = grid.size().getInfo(); +var gridList = grid.toList(gridSize); +``` Batch generate a composite image task export for each grid via looping: +```js // In each grid cell, export a composite -for (var i = 0; i < gridSize; i++) {   // Extract grid polygon and filter S2 datasets for this region.   var gridCell = ee.Feature(gridList.get(i)).geometry();   var s2Sr = ee.ImageCollection('COPERNICUS/S2_SR') -       .filterDate(start, end) -       .filterBounds(gridCell) -       .select(['B2', 'B3', 'B4', 'B5']);   var s2 = ee.ImageCollection('COPERNICUS/S2') -       .filterDate(start, end) -       .filterBounds(gridCell) -       .select(['B7', 'B8', 'B8A', 'B10']);   var s2c = ee.ImageCollection('COPERNICUS/S2_CLOUD_PROBABILITY') -       .filterDate(start, end) -       .filterBounds(gridCell);   // Apply the cloud mask.   var withCloudProbability = s2mask_tools.indexJoin(s2Sr, s2c,       'cloud_probability');   var withS2L1C = s2mask_tools.indexJoin(withCloudProbability, s2,       'l1c');   var masked = ee.ImageCollection(withS2L1C.map(s2mask_tools -       .maskImage));   // Generate a median composite and export.   var median = masked.reduce(ee.Reducer.median(), 8);   // Export.   var imagename = imageBaseName + 'tile' + i;   Export.image.toAsset({ -       image: median, -       description: imagename, -       assetId: assetCollection + '/' + imagename, -       scale: 10, -       region: gridCell, -       maxPixels: 1e13   }); +for (var i = 0; i < gridSize; i++) { // Extract grid polygon and filter S2 datasets for this region. var gridCell = ee.Feature(gridList.get(i)).geometry(); var s2Sr = ee.ImageCollection('COPERNICUS/S2_SR') + .filterDate(start, end) + .filterBounds(gridCell) + .select(['B2', 'B3', 'B4', 'B5']); var s2 = ee.ImageCollection('COPERNICUS/S2') + .filterDate(start, end) + .filterBounds(gridCell) + .select(['B7', 'B8', 'B8A', 'B10']); var s2c = ee.ImageCollection('COPERNICUS/S2_CLOUD_PROBABILITY') + .filterDate(start, end) + .filterBounds(gridCell); // Apply the cloud mask. var withCloudProbability = s2mask_tools.indexJoin(s2Sr, s2c, 'cloud_probability'); var withS2L1C = s2mask_tools.indexJoin(withCloudProbability, s2, 'l1c'); var masked = ee.ImageCollection(withS2L1C.map(s2mask_tools + .maskImage)); // Generate a median composite and export. var median = masked.reduce(ee.Reducer.median(), 8); // Export. var imagename = imageBaseName + 'tile' + i; Export.image.toAsset({ + image: median, + description: imagename, + assetId: assetCollection + '/' + imagename, + scale: 10, + region: gridCell, + maxPixels: 1e13 }); } -::: {.callout-note} -Code Checkpoint F62d. The book’s repository contains a script that shows what your code should look like at this point. +:::{.callout-note} +Code Checkpoint F62d. The book’s repository contains a script that shows what your code should look like at this point. ::: -Similar to Sect. 1.2, we now have a list of tasks to execute. We can hold the Cmd/Ctrl key and click Run to execute each task (Fig. F6.2.9). Again, users with applications requiring large batches may want to explore the Earth Engine Python API, which is well-suited to batching work. The output ImageCollection is 35.3 GB, so you may not want to execute all (or any) of these tasks but can access our pre-generated image, as discussed below. +Similar to Sect. 1.2, we now have a list of tasks to execute. We can hold the Cmd/Ctrl key and click Run to execute each task (Fig. F6.2.9). Again, users with applications requiring large batches may want to explore the Earth Engine Python API, which is well-suited to batching work. The output ImageCollection is 35.3 GB, so you may not want to execute all (or any) of these tasks but can access our pre-generated image, as discussed below. -![](F6/image22.png) +![Fig. F6.2.9 Spatial batch tasks have been generated and are ready to run](F6/image22.png) -Fig. F6.2.9 Spatial batch tasks have been generated and are ready to run In addition to being necessary for very large regions, batch processing can speed things up for moderate scales. In our tests, tiles averaged about one hour to complete. Because three jobs in our queue were running simultaneously, we covered the full state of Washington in about four hours (compared to about nine hours when tested for the full state of Washington at once). Users should note, however, that there is also an overhead to spinning up each batch task. Finding the balance between task size and task number is a challenge for most Earth Engine users that becomes easier with experience. -In a new script, load the exported ImageCollection and mosaic for use. +In a new script, load the exported ImageCollection and mosaic for use. // load image collection and mosaic into single image -var assetCollection = 'projects/gee-book/assets/F6-2/s2_composite_WA'; -var composite = ee.ImageCollection(assetCollection).mosaic(); +var assetCollection = 'projects/gee-book/assets/F6-2/s2_composite_WA'; +var composite = ee.ImageCollection(assetCollection).mosaic(); // Display the results -var geometry = ee.Geometry.Point([-120.5873563817392,   47.39035206888694 +var geometry = ee.Geometry.Point([-120.5873563817392, 47.39035206888694 ]); Map.centerObject(geometry, 6); -var vizParams = { -   bands: ['B4_median', 'B3_median', 'B2_median'], -   min: 0, -   max: 3000 +var vizParams = { + bands: ['B4_median', 'B3_median', 'B2_median'], + min: 0, + max: 3000 }; Map.addLayer(composite, vizParams, 'median'); -::: {.callout-note} -Code Checkpoint F62e. The book’s repository contains a script that shows what your code should look like at this point. +``` +:::{.callout-note} +Code Checkpoint F62e. The book’s repository contains a script that shows what your code should look like at this point. ::: -![](F6/image16.png) +![Fig. F6.2.10 Sentinel-2 composite covering the state of Washington, loaded from asset. The remaining white colors are snow-capped mountains, not clouds.](F6/image16.png) -Fig. F6.2.10 Sentinel-2 composite covering the state of Washington, loaded from asset. The remaining white colors are snow-capped mountains, not clouds. Note the ease, speed, and joy of panning and zooming to explore the pre-computed composite asset (Fig. F6.2.10) compared to the on-the-fly version discussed in Sect. 2.1. @@ -1661,19 +1695,17 @@ Here, we provide tips for managing multipart workflows. These are somewhat opini Tip 1. Create a repository for each project -The repository can be considered the fundamental project unit. In Earth Engine, sharing permissions are set for each individual repository, so this allows you to share a specific project with others (see Chap. F6.1). +The repository can be considered the fundamental project unit. In Earth Engine, sharing permissions are set for each individual repository, so this allows you to share a specific project with others (see Chap. F6.1). -By default, Earth Engine saves new scripts in a “default” repository specific for each user (users//default). You can create new repositories on the Scripts tab of the Code Editor (Fig. F6.2.11). +By default, Earth Engine saves new scripts in a “default” repository specific for each user (users//default). You can create new repositories on the Scripts tab of the Code Editor (Fig. F6.2.11). -![](F6/image20.png) +![Fig. F6.2.11 The Code Editor menu for creating new repositories](F6/image20.png) -Fig. F6.2.11 The Code Editor menu for creating new repositories To adjust permissions for each repository, click on the Gear icon (Fig. F6.2.12): -![](F6/image67.png) +![Fig. F6.2.12 Access the sharing and permissions menu for each repository by clicking the Gear icon](F6/image67.png) -Fig. F6.2.12 Access the sharing and permissions menu for each repository by clicking the Gear icon For users familiar with version control, Earth Engine uses a git-based script manager, so each repository can also be managed, edited, and/or synced with your local copy or collaborative spaces like GitHub. @@ -1681,9 +1713,8 @@ Tip 2. Make a separate script for each step, and make script file names informat Descriptive, self-sorting filenames are an excellent “good enough” way to keep your projects organized. We recommend starting script names with zero-padded numeric values to take advantage of default ordering. Because we are generating assets in early scripts that are used in later scripts, it’s important to preserve the order of your workflow. The name should also include short descriptions of what the script does (Fig. F6.2.13). -![](F6/image62.png) +![Fig. F6.2.13 An example project repository with multiple scripts. Using leading numbers when naming scripts allows you to order them by their position in the workflow.](F6/image62.png) -Fig. F6.2.13 An example project repository with multiple scripts. Using leading numbers when naming scripts allows you to order them by their position in the workflow. Leaving some decimal places between successive scripts gives you the ability to easily insert any additional steps you didn’t originally anticipate. And zero-padding means your self-sorting still works once you move into double-digit numbers. @@ -1693,15 +1724,15 @@ Tip 3. Consider data types and file sizes when storing intermediates Images and image collections are common intermediate file types, since generating satellite composites or creating land use classifications tends to be computationally intensive. These assets can also be quite large, depending on the resolution and region size. Recall that our single-year, subregional Sentinel-2 composite in Sect. 2 was about 23 GB. -Image values can be stored from 8-bit integers to 64-bit double floats (numbers with decimals). Higher bits allow for more precision, but have much larger file sizes and are not always necessary. For example, if we are generating a land use map with five classes, we can convert that to a signed or unsigned 8-bit integer using toInt8 or toUint8 prior to exporting to asset, which can accommodate 256 unique values. This results in a smaller file size. Selectively retaining only bands of interest is also helpful to reduce size. +Image values can be stored from 8-bit integers to 64-bit double floats (numbers with decimals). Higher bits allow for more precision, but have much larger file sizes and are not always necessary. For example, if we are generating a land use map with five classes, we can convert that to a signed or unsigned 8-bit integer using toInt8 or toUint8 prior to exporting to asset, which can accommodate 256 unique values. This results in a smaller file size. Selectively retaining only bands of interest is also helpful to reduce size. -For cases requiring decimals and precision, consider whether a 32-bit float will do the job instead of a 64-bit double—toFloat will convert an image to a 32-bit float. If you find you need to conserve storage, you can also scale float values and store as an integer image (image.multiply(100).toInt16(), for example). This would retain precision to the second decimal place and reduce file size by a factor of two. Note that this may require you to unscale the values in downstream use. Ultimately, the appropriate data type will be specific to your needs. +For cases requiring decimals and precision, consider whether a 32-bit float will do the job instead of a 64-bit double—toFloat will convert an image to a 32-bit float. If you find you need to conserve storage, you can also scale float values and store as an integer image (image.multiply(100).toInt16(), for example). This would retain precision to the second decimal place and reduce file size by a factor of two. Note that this may require you to unscale the values in downstream use. Ultimately, the appropriate data type will be specific to your needs. And of course, as mentioned above under “The Importance of Best Coding Practices,” be aware of the scale resolution you are working at, and avoid using unnecessarily high resolution when it’s not supported by either the input imagery or your research goals. Tip 4. Consider Google Cloud Platform for hosting larger intermediates -If you are working with very large or very many files, you can link Earth Engine with Cloud Projects on Google Cloud Platform. See the Earth Engine documentation on “Setting Up Earth Engine Enabled Cloud Projects” for more information. +If you are working with very large or very many files, you can link Earth Engine with Cloud Projects on Google Cloud Platform. See the Earth Engine documentation on “Setting Up Earth Engine Enabled Cloud Projects” for more information. ## Synthesis {.unnumbered} @@ -1725,31 +1756,31 @@ Wilson G, Bryan J, Cranston K, et al (2017) Good enough practices in scientific -::: {.callout-tip} +:::{.callout-tip} # Chapter Information ## Author {.unlisted .unnumbered} -Qiusheng Wu +Qiusheng Wu ## Overview {.unlisted .unnumbered} -The purpose of this chapter is to demonstrate how to design and publish Earth Engine Apps using both JavaScript and Python. You will be introduced to the Earth Engine User Interface JavaScript API and the geemap Python package. Upon completion of this chapter, you will be able to publish an Earth Engine App with a split-panel map for visualizing land cover change. +The purpose of this chapter is to demonstrate how to design and publish Earth Engine Apps using both JavaScript and Python. You will be introduced to the Earth Engine User Interface JavaScript API and the geemap Python package. Upon completion of this chapter, you will be able to publish an Earth Engine App with a split-panel map for visualizing land cover change. ## Learning Outcomes {.unlisted .unnumbered} -* Designing a user interface for an Earth Engine App using JavaScript. +* Designing a user interface for an Earth Engine App using JavaScript. * Publishing an Earth Engine App for visualizing land cover change. * Developing an Earth Engine App using Python and geemap. * Deploying an Earth Engine App using a local computer as a web server. * Publishing an Earth Engine App using Python and free cloud platforms. -* Creating a conda environment using Anaconda/Miniconda. +* Creating a conda environment using Anaconda/Miniconda. * Installing Python packages and using Jupyter Notebook. * Commiting changes to a GitHub repository. @@ -1759,196 +1790,198 @@ The purpose of this chapter is to demonstrate how to design and publish Earth E * Import images and image collections, filter, and visualize (Part F1). * Use the basic functions and logic of Python. -::: +::: ## Introduction {.unlisted .unnumbered} -Earth Engine has a user interface API that allows users to build and publish interactive web apps directly from the JavaScript Code Editor. Many readers will have encountered a call to ui.Chart in other chapters, but much more interface functionality is available. In particular, users can utilize the ui functions to construct an entire graphical user interface (GUI) for their Earth Engine script. The GUI may include simple widgets (e.g., labels, buttons, checkboxes, sliders, text boxes) as well as more complex widgets (e.g., charts, maps, panels) for controlling the GUI layout. A complete list of the ui widgets and more information about panels can be found at the links below. Once a GUI is constructed, users can publish the App from the JavaScript Code Editor by clicking the Apps button above the script panel in the Code Editor. +Earth Engine has a user interface API that allows users to build and publish interactive web apps directly from the JavaScript Code Editor. Many readers will have encountered a call to ui.Chart in other chapters, but much more interface functionality is available. In particular, users can utilize the ui functions to construct an entire graphical user interface (GUI) for their Earth Engine script. The GUI may include simple widgets (e.g., labels, buttons, checkboxes, sliders, text boxes) as well as more complex widgets (e.g., charts, maps, panels) for controlling the GUI layout. A complete list of the ui widgets and more information about panels can be found at the links below. Once a GUI is constructed, users can publish the App from the JavaScript Code Editor by clicking the Apps button above the script panel in the Code Editor. * Widgets: [https://developers.google.com/earth-engine/guides/ui_widgets](https://www.google.com/url?q=https://developers.google.com/earth-engine/guides/ui_widgets&sa=D&source=editors&ust=1671458841273029&usg=AOvVaw10aLP4KU7kHJTwcnM5Pr4-) -* Panels: [https://developers.google.com/earth-engine/guides/ui_panels](https://www.google.com/url?q=https://developers.google.com/earth-engine/guides/ui_panels&sa=D&source=editors&ust=1671458841273501&usg=AOvVaw1XwxHFyCULBnago_-VpDUq)  +* Panels: [https://developers.google.com/earth-engine/guides/ui_panels](https://www.google.com/url?q=https://developers.google.com/earth-engine/guides/ui_panels&sa=D&source=editors&ust=1671458841273501&usg=AOvVaw1XwxHFyCULBnago_-VpDUq) -Unlike the Earth Engine JavaScript API, the Earth Engine Python API does not provide functionality for building interactive user interfaces. Fortunately, the Jupyter ecosystem has ipywidgets, an architecture for creating interactive user interface controls (e.g., buttons, sliders, checkboxes, text boxes, dropdown lists) in Jupyter notebooks that communicate with Python code. The integration of graphical widgets into the Jupyter Notebook workflow allows users to configure ad hoc control panels to interactively sweep over parameters using graphical widget controls. One very powerful widget is the output widget, which can be used to display rich output generated by IPython, such as text, images, charts, and videos. A complete list of widgets and more information about the output widget can be found at the links below. By integrating ipyleaflet (for creating interactive maps) and ipywidgets (for designing interactive user interfaces), the geemap Python package ([https://geemap.org](https://www.google.com/url?q=https://geemap.org&sa=D&source=editors&ust=1671458841274245&usg=AOvVaw3rVUS8lKoS9clb8r42oxe0)) makes it much easier to explore and analyze massive Earth Engine datasets via a web browser in a Jupyter environment suitable for interactive exploration, teaching, and sharing. Users can build interactive Earth Engine Apps using geemap with minimal coding (Fig. F6.3.1). +Unlike the Earth Engine JavaScript API, the Earth Engine Python API does not provide functionality for building interactive user interfaces. Fortunately, the Jupyter ecosystem has ipywidgets, an architecture for creating interactive user interface controls (e.g., buttons, sliders, checkboxes, text boxes, dropdown lists) in Jupyter notebooks that communicate with Python code. The integration of graphical widgets into the Jupyter Notebook workflow allows users to configure ad hoc control panels to interactively sweep over parameters using graphical widget controls. One very powerful widget is the output widget, which can be used to display rich output generated by IPython, such as text, images, charts, and videos. A complete list of widgets and more information about the output widget can be found at the links below. By integrating ipyleaflet (for creating interactive maps) and ipywidgets (for designing interactive user interfaces), the geemap Python package ([https://geemap.org](https://www.google.com/url?q=https://geemap.org&sa=D&source=editors&ust=1671458841274245&usg=AOvVaw3rVUS8lKoS9clb8r42oxe0)) makes it much easier to explore and analyze massive Earth Engine datasets via a web browser in a Jupyter environment suitable for interactive exploration, teaching, and sharing. Users can build interactive Earth Engine Apps using geemap with minimal coding (Fig. F6.3.1). -* Widgets: [https://ipywidgets.readthedocs.io/en/latest/examples/Widget%20List.html](https://www.google.com/url?q=https://ipywidgets.readthedocs.io/en/latest/examples/Widget%2520List.html&sa=D&source=editors&ust=1671458841274780&usg=AOvVaw2SzMGW9F4r3E-llbiCksCA)  -* Output: [https://ipywidgets.readthedocs.io/en/latest/examples/Output%20Widget.html](https://www.google.com/url?q=https://ipywidgets.readthedocs.io/en/latest/examples/Output%2520Widget.html&sa=D&source=editors&ust=1671458841275255&usg=AOvVaw13a3qsly2f0mg0egoTAMqw)  +* Widgets: [https://ipywidgets.readthedocs.io/en/latest/examples/Widget%20List.html](https://www.google.com/url?q=https://ipywidgets.readthedocs.io/en/latest/examples/Widget%2520List.html&sa=D&source=editors&ust=1671458841274780&usg=AOvVaw2SzMGW9F4r3E-llbiCksCA) +* Output: [https://ipywidgets.readthedocs.io/en/latest/examples/Output%20Widget.html](https://www.google.com/url?q=https://ipywidgets.readthedocs.io/en/latest/examples/Output%2520Widget.html&sa=D&source=editors&ust=1671458841275255&usg=AOvVaw13a3qsly2f0mg0egoTAMqw) -![](F6/image1.png) +![Fig. F6.3.1 The GUI of geemap in a Jupyter environment](F6/image1.png) -Fig. F6.3.1 The GUI of geemap in a Jupyter environment ## Building an Earth Engine App Using JavaScript -In this section, you will learn how to design a user interface for an Earth Engine App using JavaScript and the Earth Engine User Interface API. Upon completion of this section, you will have an Earth Engine App with a split-panel map for visualizing land cover change using the Landsat-based United States Geological Survey National Land Cover Database (NLCD). +In this section, you will learn how to design a user interface for an Earth Engine App using JavaScript and the Earth Engine User Interface API. Upon completion of this section, you will have an Earth Engine App with a split-panel map for visualizing land cover change using the Landsat-based United States Geological Survey National Land Cover Database (NLCD). -First, let’s define a function for filtering the NLCD ImageCollection by year and select the landcover band. The function returns an Earth Engine ui.Map.Layer of the landcover band of the selected NLCD image. Note that as of this writing, NLCD spans nine epochs: 1992, 2001, 2004, 2006, 2008, 2011, 2013, 2016, and 2019. The 1992 data are primarily based on unsupervised classification of Landsat data, while the rest of the images rely on the imperviousness data layer for the urban classes and on a decision-tree classification for the rest. The 1992 image is not directly comparable to any later editions of NLCD (see the Earth Engine Data Catalog for more details, if needed). Therefore, we will use only the eight epochs after 2000 in this lab.   +First, let’s define a function for filtering the NLCD ImageCollection by year and select the landcover band. The function returns an Earth Engine ui.Map.Layer of the landcover band of the selected NLCD image. Note that as of this writing, NLCD spans nine epochs: 1992, 2001, 2004, 2006, 2008, 2011, 2013, 2016, and 2019. The 1992 data are primarily based on unsupervised classification of Landsat data, while the rest of the images rely on the imperviousness data layer for the urban classes and on a decision-tree classification for the rest. The 1992 image is not directly comparable to any later editions of NLCD (see the Earth Engine Data Catalog for more details, if needed). Therefore, we will use only the eight epochs after 2000 in this lab. +```js // Get an NLCD image by year. -var getNLCD = function(year) {   // Import the NLCD collection.   var dataset = ee.ImageCollection(       'USGS/NLCD_RELEASES/2019_REL/NLCD');   // Filter the collection by year.   var nlcd = dataset.filter(ee.Filter.eq('system:index', year)) -       .first();   // Select the land cover band.   var landcover = nlcd.select('landcover');   return ui.Map.Layer(landcover, {}, year); +var getNLCD = function(year) { // Import the NLCD collection. var dataset = ee.ImageCollection( 'USGS/NLCD_RELEASES/2019_REL/NLCD'); // Filter the collection by year. var nlcd = dataset.filter(ee.Filter.eq('system:index', year)) + .first(); // Select the land cover band. var landcover = nlcd.select('landcover'); return ui.Map.Layer(landcover, {}, year); }; -Our intention is to create a dropdown list so that when a particular epoch is selected, the corresponding NLCD image layer will be displayed on the map. We’ll define a dictionary with each NLCD epoch as the key and its corresponding NLCD image layer as the value. The keys of the dictionary (i.e., the eight NLCD epochs) will be used as the input to the dropdown lists (ui.Select) on the split-level map. +``` +Our intention is to create a dropdown list so that when a particular epoch is selected, the corresponding NLCD image layer will be displayed on the map. We’ll define a dictionary with each NLCD epoch as the key and its corresponding NLCD image layer as the value. The keys of the dictionary (i.e., the eight NLCD epochs) will be used as the input to the dropdown lists (ui.Select) on the split-level map. +```js // Create a dictionary with each year as the key // and its corresponding NLCD image layer as the value. -var images = {   '2001': getNLCD('2001'),   '2004': getNLCD('2004'),   '2006': getNLCD('2006'),   '2008': getNLCD('2008'),   '2011': getNLCD('2011'),   '2013': getNLCD('2013'),   '2016': getNLCD('2016'),   '2019': getNLCD('2019'), +var images = { '2001': getNLCD('2001'), '2004': getNLCD('2004'), '2006': getNLCD('2006'), '2008': getNLCD('2008'), '2011': getNLCD('2011'), '2013': getNLCD('2013'), '2016': getNLCD('2016'), '2019': getNLCD('2019'), }; -The split-panel map is composed of two individual maps, leftMap and rightMap. The map controls (e.g., zoomControl, scaleControl, mapTypeControl) will be shown only on rightMap. A control panel (ui.Panel) composed of a label (ui.Label) and a dropdown list (ui.Select) is added to each map. When an NLCD epoch is selected from a dropdown list, the function updateMap will be called to show the corresponding image layer of the selected epoch. +``` +The split-panel map is composed of two individual maps, leftMap and rightMap. The map controls (e.g., zoomControl, scaleControl, mapTypeControl) will be shown only on rightMap. A control panel (ui.Panel) composed of a label (ui.Label) and a dropdown list (ui.Select) is added to each map. When an NLCD epoch is selected from a dropdown list, the function updateMap will be called to show the corresponding image layer of the selected epoch. +```js // Create the left map, and have it display the first layer. -var leftMap = ui.Map(); +var leftMap = ui.Map(); leftMap.setControlVisibility(false); -var leftSelector = addLayerSelector(leftMap, 0, 'top-left'); +var leftSelector = addLayerSelector(leftMap, 0, 'top-left'); // Create the right map, and have it display the last layer. -var rightMap = ui.Map(); +var rightMap = ui.Map(); rightMap.setControlVisibility(true); -var rightSelector = addLayerSelector(rightMap, 7, 'top-right'); +var rightSelector = addLayerSelector(rightMap, 7, 'top-right'); // Adds a layer selection widget to the given map, to allow users to // change which image is displayed in the associated map. -function addLayerSelector(mapToChange, defaultValue, position) {   var label = ui.Label('Select a year:');   // This function changes the given map to show the selected image.   function updateMap(selection) { -       mapToChange.layers().set(0, images[selection]); -   }   // Configure a selection dropdown to allow the user to choose   // between images, and set the map to update when a user   // makes a selection.   var select = ui.Select({ -       items: Object.keys(images), -       onChange: updateMap -   }); -   select.setValue(Object.keys(images)[defaultValue], true);   var controlPanel =       ui.Panel({ -           widgets: [label, select], -           style: { -               position: position -           } -       }); +function addLayerSelector(mapToChange, defaultValue, position) { var label = ui.Label('Select a year:'); // This function changes the given map to show the selected image. function updateMap(selection) { + mapToChange.layers().set(0, images[selection]); + } // Configure a selection dropdown to allow the user to choose // between images, and set the map to update when a user // makes a selection. var select = ui.Select({ + items: Object.keys(images), + onChange: updateMap + }); + select.setValue(Object.keys(images)[defaultValue], true); var controlPanel = ui.Panel({ + widgets: [label, select], + style: { + position: position + } + }); -   mapToChange.add(controlPanel); + mapToChange.add(controlPanel); } -When displaying a land cover classification image on the Map, it would be useful to add a legend to make it easier for users to interpret the land cover type associated with each color. Let’s define a dictionary that will be used to construct the legend. The dictionary contains two keys: names (a list of land cover types) and colors (a list of colors associated with each land cover type). The legend will be placed in the bottom right of the Map.     +When displaying a land cover classification image on the Map, it would be useful to add a legend to make it easier for users to interpret the land cover type associated with each color. Let’s define a dictionary that will be used to construct the legend. The dictionary contains two keys: names (a list of land cover types) and colors (a list of colors associated with each land cover type). The legend will be placed in the bottom right of the Map. // Set the legend title. -var title = 'NLCD Land Cover Classification'; +var title = 'NLCD Land Cover Classification'; // Set the legend position. -var position = 'bottom-right'; +var position = 'bottom-right'; // Define a dictionary that will be used to make a legend -var dict = {   'names': [       '11        Open Water',       '12        Perennial Ice/Snow',       '21        Developed, Open Space',       '22        Developed, Low Intensity',       '23        Developed, Medium Intensity',       '24        Developed, High Intensity',       '31        Barren Land (Rock/Sand/Clay)',       '41        Deciduous Forest',       '42        Evergreen Forest',       '43        Mixed Forest',       '51        Dwarf Scrub',       '52        Shrub/Scrub',       '71        Grassland/Herbaceous',       '72        Sedge/Herbaceous',       '73        Lichens',       '74        Moss',       '81        Pasture/Hay',       '82        Cultivated Crops',       '90        Woody Wetlands',       '95        Emergent Herbaceous Wetlands', -   ],   'colors': [       '#466b9f', '#d1def8', '#dec5c5', '#d99282', '#eb0000',       '#ab0000',       '#b3ac9f', '#68ab5f', '#1c5f2c', '#b5c58f', '#af963c',       '#ccb879',       '#dfdfc2', '#d1d182', '#a3cc51', '#82ba9e', '#dcd939',       '#ab6c28',       '#b8d9eb', '#6c9fb8', -   ] +var dict = { 'names': [ '11 Open Water', '12 Perennial Ice/Snow', '21 Developed, Open Space', '22 Developed, Low Intensity', '23 Developed, Medium Intensity', '24 Developed, High Intensity', '31 Barren Land (Rock/Sand/Clay)', '41 Deciduous Forest', '42 Evergreen Forest', '43 Mixed Forest', '51 Dwarf Scrub', '52 Shrub/Scrub', '71 Grassland/Herbaceous', '72 Sedge/Herbaceous', '73 Lichens', '74 Moss', '81 Pasture/Hay', '82 Cultivated Crops', '90 Woody Wetlands', '95 Emergent Herbaceous Wetlands', + ], 'colors': [ '#466b9f', '#d1def8', '#dec5c5', '#d99282', '#eb0000', '#ab0000', '#b3ac9f', '#68ab5f', '#1c5f2c', '#b5c58f', '#af963c', '#ccb879', '#dfdfc2', '#d1d182', '#a3cc51', '#82ba9e', '#dcd939', '#ab6c28', '#b8d9eb', '#6c9fb8', + ] }; +``` With the legend dictionary defined above, we can now create a panel to hold the legend widget and add it to the Map. Each row on the legend widget is composed of a color box followed by its corresponding land cover type. +```js // Create a panel to hold the legend widget. -var legend = ui.Panel({ -   style: { -       position: position, -       padding: '8px 15px'   } +var legend = ui.Panel({ + style: { + position: position, + padding: '8px 15px' } });// Function to generate the legend. -function addCategoricalLegend(panel, dict, title) {   // Create and add the legend title.   var legendTitle = ui.Label({ -       value: title, -       style: { -           fontWeight: 'bold', -           fontSize: '18px', -           margin: '0 0 4px 0', -           padding: '0'       } -   }); -   panel.add(legendTitle);   var loading = ui.Label('Loading legend...', { -       margin: '2px 0 4px 0'   }); -   panel.add(loading);   // Creates and styles 1 row of the legend.   var makeRow = function(color, name) {       // Create the label that is actually the colored box.       var colorBox = ui.Label({ -           style: { -               backgroundColor: color,               // Use padding to give the box height and width.               padding: '8px', -               margin: '0 0 4px 0'           } -       });       // Create the label filled with the description text.       var description = ui.Label({ -           value: name, -           style: { -               margin: '0 0 4px 6px'           } -       });       return ui.Panel({ -           widgets: [colorBox, description], -           layout: ui.Panel.Layout.Flow('horizontal') -       }); -   };   // Get the list of palette colors and class names from the image.   var palette = dict.colors;   var names = dict.names; -   loading.style().set('shown', false);   for (var i = 0; i < names.length; i++) { -       panel.add(makeRow(palette[i], names[i])); -   } +function addCategoricalLegend(panel, dict, title) { // Create and add the legend title. var legendTitle = ui.Label({ + value: title, + style: { + fontWeight: 'bold', + fontSize: '18px', + margin: '0 0 4px 0', + padding: '0' } + }); + panel.add(legendTitle); var loading = ui.Label('Loading legend...', { + margin: '2px 0 4px 0' }); + panel.add(loading); // Creates and styles 1 row of the legend. var makeRow = function(color, name) { // Create the label that is actually the colored box. var colorBox = ui.Label({ + style: { + backgroundColor: color, // Use padding to give the box height and width. padding: '8px', + margin: '0 0 4px 0' } + }); // Create the label filled with the description text. var description = ui.Label({ + value: name, + style: { + margin: '0 0 4px 6px' } + }); return ui.Panel({ + widgets: [colorBox, description], + layout: ui.Panel.Layout.Flow('horizontal') + }); + }; // Get the list of palette colors and class names from the image. var palette = dict.colors; var names = dict.names; + loading.style().set('shown', false); for (var i = 0; i < names.length; i++) { + panel.add(makeRow(palette[i], names[i])); + } -   rightMap.add(panel); + rightMap.add(panel); } -The last step is to create a split-panel map to hold the linked maps (leftMap and rightMap) and tie everything together. When users pan and zoom one map, the other map will also be panned and zoomed to the same extent automatically. When users select a year from a dropdown list, the image layer will be updated accordingly. Users can use the slider to swipe through and visualize land cover change easily (Fig. F6.3.2). Please make sure you minimize the Code Editor and maximize the Map so that you can see the dropdown widget in the upper-right corner of the map. +The last step is to create a split-panel map to hold the linked maps (leftMap and rightMap) and tie everything together. When users pan and zoom one map, the other map will also be panned and zoomed to the same extent automatically. When users select a year from a dropdown list, the image layer will be updated accordingly. Users can use the slider to swipe through and visualize land cover change easily (Fig. F6.3.2). Please make sure you minimize the Code Editor and maximize the Map so that you can see the dropdown widget in the upper-right corner of the map. addCategoricalLegend(legend, dict, title); // Create a SplitPanel to hold the adjacent, linked maps. -var splitPanel = ui.SplitPanel({ -   firstPanel: leftMap, -   secondPanel: rightMap, -   wipe: true, -   style: { -       stretch: 'both'   } +var splitPanel = ui.SplitPanel({ + firstPanel: leftMap, + secondPanel: rightMap, + wipe: true, + style: { + stretch: 'both' } });// Set the SplitPanel as the only thing in the UI root. ui.root.widgets().reset([splitPanel]); -var linker = ui.Map.Linker([leftMap, rightMap]); +var linker = ui.Map.Linker([leftMap, rightMap]); leftMap.setCenter(-100, 40, 4); -::: {.callout-note} -Code Checkpoint F63a. The book’s repository contains a script that shows what your code should look like at this point. +``` +:::{.callout-note} +Code Checkpoint F63a. The book’s repository contains a script that shows what your code should look like at this point. ::: -![](F6/image32.png) +![Fig. F6.3.2 A split-panel map for visualizing land cover change using NLCD](F6/image32.png) -Fig. F6.3.2 A split-panel map for visualizing land cover change using NLCD ## Publishing an Earth Engine App from the Code Editor -The goal of this section is to publish the Earth Engine App that we created in Sect. 1. The look and feel of interfaces changes often; if the exact windows described below change over time, the concepts should remain stable to help you to publish your App. First, load the script (see the Code Checkpoint in Sect. 1) into the Code Editor. Then, open the Manage Apps panel by clicking the Apps button above the script panel in the Code Editor (Fig. F6.3.3). +The goal of this section is to publish the Earth Engine App that we created in Sect. 1. The look and feel of interfaces changes often; if the exact windows described below change over time, the concepts should remain stable to help you to publish your App. First, load the script (see the Code Checkpoint in Sect. 1) into the Code Editor. Then, open the Manage Apps panel by clicking the Apps button above the script panel in the Code Editor (Fig. F6.3.3). -![](F6/image14.png) +![Fig. F6.3.3 The Apps button in the JavaScript Code Editor](F6/image14.png) -Fig. F6.3.3 The Apps button in the JavaScript Code Editor -Now click on the New App button (Fig. F6.3.4). +Now click on the New App button (Fig. F6.3.4). -![](F6/image61.png) +![Fig. F6.3.4 The New App button](F6/image61.png) -Fig. F6.3.4 The New App button -In the Publish New App dialog (Fig. F6.3.5), choose a name for the App (e.g., NLCD Land Cover Change), select a Google Cloud Project, provide a thumbnail to be shown in the Public Apps Gallery, and specify the location of the App’s source code. You may restrict access to the App to a particular Google Group or make it publicly accessible. Check Feature this app in your Public Apps Gallery if you would like this App to appear in your public gallery of Apps available at https://USERNAME.users.earthengine.app. When all fields are filled out and validated, the Publish button will be enabled; click it to complete publishing the App.  +In the Publish New App dialog (Fig. F6.3.5), choose a name for the App (e.g., NLCD Land Cover Change), select a Google Cloud Project, provide a thumbnail to be shown in the Public Apps Gallery, and specify the location of the App’s source code. You may restrict access to the App to a particular Google Group or make it publicly accessible. Check Feature this app in your Public Apps Gallery if you would like this App to appear in your public gallery of Apps available at https://USERNAME.users.earthengine.app. When all fields are filled out and validated, the Publish button will be enabled; click it to complete publishing the App. -![](F6/image37.png) +![Fig. F6.3.5 The Publish New App dialog](F6/image37.png) -Fig. F6.3.5 The Publish New App dialog -To manage an App from the Code Editor, open the Manage Apps panel (Fig. F6.3.6) by clicking the Apps button above the script panel in the Code Editor (Fig. F6.3.3). There, you can update your App’s configuration or delete the App. +To manage an App from the Code Editor, open the Manage Apps panel (Fig. F6.3.6) by clicking the Apps button above the script panel in the Code Editor (Fig. F6.3.3). There, you can update your App’s configuration or delete the App. -![](F6/image65.png) +![Fig. F6.3.6 The Manage Apps panel](F6/image65.png) -Fig. F6.3.6 The Manage Apps panel ## Developing an Earth Engine App Using geemap -In this section, you will learn how to develop an Earth Engine App using the geemap Python package and Jupyter Notebook. The geemap package is available on both [PyPI](https://www.google.com/url?q=https://pypi.org/project/leafmap&sa=D&source=editors&ust=1671458841303040&usg=AOvVaw25Lg4yxDI4b31VJovOcUOc) (pip) and [conda-forge](https://www.google.com/url?q=https://anaconda.org/conda-forge/leafmap&sa=D&source=editors&ust=1671458841303462&usg=AOvVaw267tbdQ3NyyHL53Jmk4W4k). It is highly recommended that you create a fresh conda environment to install geemap. +In this section, you will learn how to develop an Earth Engine App using the geemap Python package and Jupyter Notebook. The geemap package is available on both [PyPI](https://www.google.com/url?q=https://pypi.org/project/leafmap&sa=D&source=editors&ust=1671458841303040&usg=AOvVaw25Lg4yxDI4b31VJovOcUOc) (pip) and [conda-forge](https://www.google.com/url?q=https://anaconda.org/conda-forge/leafmap&sa=D&source=editors&ust=1671458841303462&usg=AOvVaw267tbdQ3NyyHL53Jmk4W4k). It is highly recommended that you create a fresh conda environment to install geemap. -::: {.callout-note} -Code Checkpoint F63b. The book’s repository contains information about setting up a conda environment and installing geemap. +:::{.callout-note} +Code Checkpoint F63b. The book’s repository contains information about setting up a conda environment and installing geemap. ::: Once you have launched a Jupyter notebook in your browser, you can continue with the next steps of the lab. -On the Jupyter Notebook interface, click the New button in the upper-right corner and select Python 3 to create a new notebook. Change the name of the notebook from “Untitled” to something meaningful (e.g., “nlcd_app”). With the newly created notebook, we can start writing and executing Python code. +On the Jupyter Notebook interface, click the New button in the upper-right corner and select Python 3 to create a new notebook. Change the name of the notebook from “Untitled” to something meaningful (e.g., “nlcd_app”). With the newly created notebook, we can start writing and executing Python code. -First, let’s import the Earth Engine and geemap libraries. Press Alt + Enter to execute the code and create a new cell below. +First, let’s import the Earth Engine and geemap libraries. Press Alt + Enter to execute the code and create a new cell below. -import eeimport geemap +import eeimport geemap -Create an interactive map by specifying the map center (latitude, longitude) and zoom level (1–18). If this is the first time you use geemap and the Earth Engine Python API, a new tab will open in your browser asking you to authenticate Earth Engine. Follow the on-screen instructions to authenticate Earth Engine. +Create an interactive map by specifying the map center (latitude, longitude) and zoom level (1–18). If this is the first time you use geemap and the Earth Engine Python API, a new tab will open in your browser asking you to authenticate Earth Engine. Follow the on-screen instructions to authenticate Earth Engine. Map = geemap.Map(center=[40, -100], zoom=4) Map -Retrieve the NLCD 2019 image by filtering the NLCD ImageCollection and selecting the landcover band. Display the NLCD 2019 image on the interactive Map by using Map.addLayer. +Retrieve the NLCD 2019 image by filtering the NLCD ImageCollection and selecting the landcover band. Display the NLCD 2019 image on the interactive Map by using Map.addLayer. # Import the NLCD collection. dataset = ee.ImageCollection('USGS/NLCD_RELEASES/2019_REL/NLCD') @@ -1962,24 +1995,23 @@ landcover = nlcd2019.select('landcover') # Display land cover on the map.Map.addLayer(landcover, {}, 'NLCD 2019') Map -Next, add the NLCD legend to the Map. The geemap package has several built-in legends, including the NLCD legend. Therefore, you can add the NLCD legend to the Map by using just one line of code (Map.add_legend). +Next, add the NLCD legend to the Map. The geemap package has several built-in legends, including the NLCD legend. Therefore, you can add the NLCD legend to the Map by using just one line of code (Map.add_legend). title = 'NLCD Land Cover Classification' Map.add_legend(legend_title=title, builtin_legend='NLCD') -Alternatively, if you want to add a custom legend, you can define a legend dictionary with labels as keys and colors as values, then you can use Map.add_legend to add the custom legend to the Map.   +Alternatively, if you want to add a custom legend, you can define a legend dictionary with labels as keys and colors as values, then you can use Map.add_legend to add the custom legend to the Map. -legend_dict = {   '11 Open Water': '466b9f',   '12 Perennial Ice/Snow': 'd1def8',   '21 Developed, Open Space': 'dec5c5',   '22 Developed, Low Intensity': 'd99282',   '23 Developed, Medium Intensity': 'eb0000',   '24 Developed High Intensity': 'ab0000',   '31 Barren Land (Rock/Sand/Clay)': 'b3ac9f',   '41 Deciduous Forest': '68ab5f',   '42 Evergreen Forest': '1c5f2c',   '43 Mixed Forest': 'b5c58f',   '51 Dwarf Scrub': 'af963c',   '52 Shrub/Scrub': 'ccb879',   '71 Grassland/Herbaceous': 'dfdfc2',   '72 Sedge/Herbaceous': 'd1d182',   '73 Lichens': 'a3cc51',   '74 Moss': '82ba9e',   '81 Pasture/Hay': 'dcd939',   '82 Cultivated Crops': 'ab6c28',   '90 Woody Wetlands': 'b8d9eb',   '95 Emergent Herbaceous Wetlands': '6c9fb8'} +legend_dict = { '11 Open Water': '466b9f', '12 Perennial Ice/Snow': 'd1def8', '21 Developed, Open Space': 'dec5c5', '22 Developed, Low Intensity': 'd99282', '23 Developed, Medium Intensity': 'eb0000', '24 Developed High Intensity': 'ab0000', '31 Barren Land (Rock/Sand/Clay)': 'b3ac9f', '41 Deciduous Forest': '68ab5f', '42 Evergreen Forest': '1c5f2c', '43 Mixed Forest': 'b5c58f', '51 Dwarf Scrub': 'af963c', '52 Shrub/Scrub': 'ccb879', '71 Grassland/Herbaceous': 'dfdfc2', '72 Sedge/Herbaceous': 'd1d182', '73 Lichens': 'a3cc51', '74 Moss': '82ba9e', '81 Pasture/Hay': 'dcd939', '82 Cultivated Crops': 'ab6c28', '90 Woody Wetlands': 'b8d9eb', '95 Emergent Herbaceous Wetlands': '6c9fb8'} title = 'NLCD Land Cover Classification' Map.add_legend(legend_title=title, legend_dict=legend_dict) -The Map with the NLCD 2019 image and legend should look like Fig. F6.3.7. +The Map with the NLCD 2019 image and legend should look like Fig. F6.3.7. -![](F6/image60.png) +![Fig. F6.3.7 The NLCD 2019 image layer displayed in geemap](F6/image60.png) -Fig. F6.3.7 The NLCD 2019 image layer displayed in geemap -The Map above shows only the NLCD 2019 image. To create an Earth Engine App for visualizing land cover change, we need a stack of NLCD images. Let’s print the list of system IDs of all available NLCD images. +The Map above shows only the NLCD 2019 image. To create an Earth Engine App for visualizing land cover change, we need a stack of NLCD images. Let’s print the list of system IDs of all available NLCD images. dataset.aggregate_array('system:id').getInfo() @@ -1998,15 +2030,15 @@ Select the eight NLCD epochs after 2000. years = ['2001', '2004', '2006', '2008', '2011', '2013', '2016', '2019'] -Define a function for filtering the NLCD ImageCollection by year and select the 'landcover' band. +Define a function for filtering the NLCD ImageCollection by year and select the 'landcover' band. # Get an NLCD image by year. -def getNLCD(year):   # Import the NLCD collection.   dataset = ee.ImageCollection('USGS/NLCD_RELEASES/2019_REL/NLCD')   # Filter the collection by year.   nlcd = dataset.filter(ee.Filter.eq('system:index', year)).first()    # Select the land cover band.   landcover = nlcd.select('landcover');   return landcover +def getNLCD(year): # Import the NLCD collection. dataset = ee.ImageCollection('USGS/NLCD_RELEASES/2019_REL/NLCD') # Filter the collection by year. nlcd = dataset.filter(ee.Filter.eq('system:index', year)).first() # Select the land cover band. landcover = nlcd.select('landcover'); return landcover -Create an NLCD ImageCollection to be used in the split-panel map. +Create an NLCD ImageCollection to be used in the split-panel map. # Create an NLCD image collection for the selected years. -collection = ee.ImageCollection(ee.List(years).map(lambda year: getNLCD(year))) +collection = ee.ImageCollection(ee.List(years).map(lambda year: getNLCD(year))) Print out the list of system IDs of the selected NLCD images covering the contiguous United States. @@ -2020,13 +2052,13 @@ The output should look like this. 'USGS/NLCD_RELEASES/2019_REL/NLCD/2008', 'USGS/NLCD_RELEASES/2019_REL/NLCD/2011', - 'USGS/NLCD_RELEASES/2019_REL/NLCD/2013', + 'USGS/NLCD_RELEASES/2019_REL/NLCD/2013', 'USGS/NLCD_RELEASES/2019_REL/NLCD/2016', 'USGS/NLCD_RELEASES/2019_REL/NLCD/2019'] Next, create a list of labels to populate the dropdown list. -labels = [f'NLCD {year}' for year in years] +labels = [f'NLCD {year}' for year in years] labels The output should look like this. @@ -2038,53 +2070,50 @@ The output should look like this. 'NLCD 2011', 'NLCD 2013', - 'NLCD 2016', + 'NLCD 2016', 'NLCD 2019'] -The last step is to create a split-panel map by passing the NLCD ImageCollection and list of labels to Map.ts_inspector. +The last step is to create a split-panel map by passing the NLCD ImageCollection and list of labels to Map.ts_inspector. Map.ts_inspector(left_ts=collection, right_ts=collection, left_names=labels, right_names=labels) Map The split-panel map should look like Fig. F6.3.8. -![](F6/image34.png) +![Fig. F6.3.8 A split-panel map for visualizing land cover change with geemap](F6/image34.png) -Fig. F6.3.8 A split-panel map for visualizing land cover change with geemap -To visualize land cover change, choose one NLCD image from the left dropdown list and another image from the right dropdown list, then use the slider to swipe through to visualize land cover change interactively. Click the close button in the bottom-right corner to close the split-panel map and return to the NLCD 2019 image shown in Fig. F6.3.7. +To visualize land cover change, choose one NLCD image from the left dropdown list and another image from the right dropdown list, then use the slider to swipe through to visualize land cover change interactively. Click the close button in the bottom-right corner to close the split-panel map and return to the NLCD 2019 image shown in Fig. F6.3.7. -::: {.callout-note} -Code Checkpoint F63c. The book’s repository contains information about what your code should look like at this point. +:::{.callout-note} +Code Checkpoint F63c. The book’s repository contains information about what your code should look like at this point. ::: ## Publishing an Earth Engine App Using a Local Web Server -In this section, you will learn how to deploy an Earth Engine App using a local computer as a web server. Assume that you have completed Sect. 3 and created a Jupyter notebook named nlcd_app.ipynb. First, you need to download ngrok, a program that can turn your computer into a secure web server and connect it to the ngrok cloud service, which accepts traffic on a public address. Download ngrok from [https://ngrok.com](https://www.google.com/url?q=https://ngrok.com&sa=D&source=editors&ust=1671458841326366&usg=AOvVaw20VcUaADMd_4ptInlJt5X1) and unzip it to a directory on your computer, then copy  nlcd_app.ipynb to the same directory. Open the Anaconda Prompt (on Windows) or the Terminal (on macOS/Linux) and enter the following commands. Make sure you change /path/to/ngrok/dir to your computer directory where the ngrok executable is located, e.g., ~/Downloads.   +In this section, you will learn how to deploy an Earth Engine App using a local computer as a web server. Assume that you have completed Sect. 3 and created a Jupyter notebook named nlcd_app.ipynb. First, you need to download ngrok, a program that can turn your computer into a secure web server and connect it to the ngrok cloud service, which accepts traffic on a public address. Download ngrok from [https://ngrok.com](https://www.google.com/url?q=https://ngrok.com&sa=D&source=editors&ust=1671458841326366&usg=AOvVaw20VcUaADMd_4ptInlJt5X1) and unzip it to a directory on your computer, then copy nlcd_app.ipynb to the same directory. Open the Anaconda Prompt (on Windows) or the Terminal (on macOS/Linux) and enter the following commands. Make sure you change /path/to/ngrok/dir to your computer directory where the ngrok executable is located, e.g., ~/Downloads. cd /path/to/ngrok/dir conda activate gee -voila --no-browser nlcd_app.ipynb +voila --no-browser nlcd_app.ipynb The output of the terminal should look like this. -![](F6/image38.png) +![Fig. F6.3.9 The output of the terminal running Voilà](F6/image38.png) -Fig. F6.3.9 The output of the terminal running Voilà -[Voilà](https://www.google.com/url?q=https://voila.readthedocs.io/en/stable/&sa=D&source=editors&ust=1671458841329454&usg=AOvVaw16GCXDSkxu4fUl6f6LFafy) can be used to run, convert, and serve a Jupyter notebook as a standalone app. Click the link (e.g., [http://localhost:8866](https://www.google.com/url?q=http://localhost:8866&sa=D&source=editors&ust=1671458841329837&usg=AOvVaw052EvlmJeJTl9zIHvFobLl)) shown in the terminal window to launch the interactive dashboard. Note that the port number is 8866, which is needed in the next step to launch ngrok. Open another terminal and enter the following command. +[Voilà](https://www.google.com/url?q=https://voila.readthedocs.io/en/stable/&sa=D&source=editors&ust=1671458841329454&usg=AOvVaw16GCXDSkxu4fUl6f6LFafy) can be used to run, convert, and serve a Jupyter notebook as a standalone app. Click the link (e.g., [http://localhost:8866](https://www.google.com/url?q=http://localhost:8866&sa=D&source=editors&ust=1671458841329837&usg=AOvVaw052EvlmJeJTl9zIHvFobLl)) shown in the terminal window to launch the interactive dashboard. Note that the port number is 8866, which is needed in the next step to launch ngrok. Open another terminal and enter the following command. cd /path/to/ngrok/dir ngrok http 8866 -The output of the terminal should look like Fig. F6.3.10. Click the link shown in the terminal window to launch the interactive dashboard. The link should look like https://random-string.ngrok.io, which is publicly accessible. Anyone with the link will be able to launch the interactive dashboard and use the split-panel map to visualize NLCD land cover change. Keep in mind that the dashboard might take several seconds to load, so please be patient. +The output of the terminal should look like Fig. F6.3.10. Click the link shown in the terminal window to launch the interactive dashboard. The link should look like https://random-string.ngrok.io, which is publicly accessible. Anyone with the link will be able to launch the interactive dashboard and use the split-panel map to visualize NLCD land cover change. Keep in mind that the dashboard might take several seconds to load, so please be patient. -![](F6/image8.png) +![Fig. F6.3.10 The output of the terminal running ngrok](F6/image8.png) -Fig. F6.3.10 The output of the terminal running ngrok -To stop the web server, press Ctrl + C on both terminal windows. See below for some optional settings for running Voilà and ngrok. +To stop the web server, press Ctrl + C on both terminal windows. See below for some optional settings for running Voilà and ngrok. To show code cells from your App, run the following from the terminal. @@ -2092,109 +2121,102 @@ voila --no-browser --strip_sources=False nlcd_app.ipynb To protect your App with a password, run the following. -ngrok http -auth="username:password" 8866 +ngrok http -auth="username:password" 8866 ## Publish an Earth Engine App Using Cloud Platforms -In this section, you will learn how to deploy an Earth Engine App on cloud platforms, such as Heroku and Google App Engine. Heroku is a “platform as a service” that enables developers to build, run, and operate applications entirely in the cloud. It has a free tier with limited computing hours, which would be sufficient for this lab. Follow the steps below to deploy the Earth Engine App on Heroku. +In this section, you will learn how to deploy an Earth Engine App on cloud platforms, such as Heroku and Google App Engine. Heroku is a “platform as a service” that enables developers to build, run, and operate applications entirely in the cloud. It has a free tier with limited computing hours, which would be sufficient for this lab. Follow the steps below to deploy the Earth Engine App on Heroku. -First, go to [https://github.com/signup](https://www.google.com/url?q=https://github.com/signup&sa=D&source=editors&ust=1671458841334585&usg=AOvVaw3ZgJomY_82B7WiuJEZym5g) to sign up for a GitHub account if you don’t have one already. Once your GitHub account has been created, log into your account and navigate to the sample app repository: [https://github.com/giswqs/earthengine-apps](https://www.google.com/url?q=https://github.com/giswqs/earthengine-apps&sa=D&source=editors&ust=1671458841335025&usg=AOvVaw1IQJyv6EFLU5bCmsOsYkfX). Click the Fork button in the top-right corner to fork this repository into your account. Two important files in the repository are worth mentioning here: [requirements.txt](https://www.google.com/url?q=https://github.com/giswqs/earthengine-apps/blob/master/requirements.txt&sa=D&source=editors&ust=1671458841335397&usg=AOvVaw23DFUPWYa0pVU-kSzjGeHr) lists the required packages (e.g., geemap) to run the App, while [Procfile](https://www.google.com/url?q=https://github.com/giswqs/earthengine-apps/blob/master/Procfile&sa=D&source=editors&ust=1671458841335755&usg=AOvVaw2_QbFT8c3ahut0eHTdTmUh) specifies the commands that are executed by the App on startup. The content of [Procfile](https://www.google.com/url?q=https://github.com/giswqs/earthengine-apps/blob/master/Procfile&sa=D&source=editors&ust=1671458841336035&usg=AOvVaw2MdhcFIHFTRIbZBL-XZFCb) should look like this. +First, go to [https://github.com/signup](https://www.google.com/url?q=https://github.com/signup&sa=D&source=editors&ust=1671458841334585&usg=AOvVaw3ZgJomY_82B7WiuJEZym5g) to sign up for a GitHub account if you don’t have one already. Once your GitHub account has been created, log into your account and navigate to the sample app repository: [https://github.com/giswqs/earthengine-apps](https://www.google.com/url?q=https://github.com/giswqs/earthengine-apps&sa=D&source=editors&ust=1671458841335025&usg=AOvVaw1IQJyv6EFLU5bCmsOsYkfX). Click the Fork button in the top-right corner to fork this repository into your account. Two important files in the repository are worth mentioning here: [requirements.txt](https://www.google.com/url?q=https://github.com/giswqs/earthengine-apps/blob/master/requirements.txt&sa=D&source=editors&ust=1671458841335397&usg=AOvVaw23DFUPWYa0pVU-kSzjGeHr) lists the required packages (e.g., geemap) to run the App, while [Procfile](https://www.google.com/url?q=https://github.com/giswqs/earthengine-apps/blob/master/Procfile&sa=D&source=editors&ust=1671458841335755&usg=AOvVaw2_QbFT8c3ahut0eHTdTmUh) specifies the commands that are executed by the App on startup. The content of [Procfile](https://www.google.com/url?q=https://github.com/giswqs/earthengine-apps/blob/master/Procfile&sa=D&source=editors&ust=1671458841336035&usg=AOvVaw2MdhcFIHFTRIbZBL-XZFCb) should look like this. -web: voila --port=$PORT --no-browser --strip_sources=True --enable_nbextensions=True --MappingKernelManager.cull_interval=60 --MappingKernelManager.cull_idle_timeout=120 notebooks/ +web: voila --port=$PORT --no-browser --strip_sources=True --enable_nbextensions=True --MappingKernelManager.cull_interval=60 --MappingKernelManager.cull_idle_timeout=120 notebooks/ -The above command instructs the server to hide the source code and periodically check for idle kernels—in this example, every 60 seconds—and cull them if they have been idle for more than 120 seconds. This can avoid idle kernels using up the server computing resources. The page served by Voilà will contain a list of any notebooks in the [notebooks](https://www.google.com/url?q=https://github.com/giswqs/earthengine-apps/tree/master/notebooks&sa=D&source=editors&ust=1671458841337601&usg=AOvVaw1pHGG9qzGbdG7h7tVr7C9e) directory. +The above command instructs the server to hide the source code and periodically check for idle kernels—in this example, every 60 seconds—and cull them if they have been idle for more than 120 seconds. This can avoid idle kernels using up the server computing resources. The page served by Voilà will contain a list of any notebooks in the [notebooks](https://www.google.com/url?q=https://github.com/giswqs/earthengine-apps/tree/master/notebooks&sa=D&source=editors&ust=1671458841337601&usg=AOvVaw1pHGG9qzGbdG7h7tVr7C9e) directory. -Next, go to [https://signup.heroku.com](https://www.google.com/url?q=https://signup.heroku.com&sa=D&source=editors&ust=1671458841338152&usg=AOvVaw03SbXy3optdTU9A2yG-bBq) to sign up for a Heroku account if you don’t have one already. Log into your account and click the New button in the top-right corner, then choose Create new app from the dropdown list (Fig. F6.3.11). +Next, go to [https://signup.heroku.com](https://www.google.com/url?q=https://signup.heroku.com&sa=D&source=editors&ust=1671458841338152&usg=AOvVaw03SbXy3optdTU9A2yG-bBq) to sign up for a Heroku account if you don’t have one already. Log into your account and click the New button in the top-right corner, then choose Create new app from the dropdown list (Fig. F6.3.11). -![](F6/image5.png) +![Fig. F6.3.11 Creating a new app in Heroku](F6/image5.png) -Fig. F6.3.11 Creating a new app in Heroku Choose a name for your App (Fig. F6.3.12). Note that the App name must be unique; if an App name has already been taken, you won’t be able to use it. -![](F6/image73.png) +![Fig. F6.3.12 Choosing an App name](F6/image73.png) -Fig. F6.3.12 Choosing an App name -Once the Heroku App has been created, click the Deploy tab and choose GitHub as the deployment method. Connect to your GitHub account and enter earthengine-apps in the search box. The repository should be listed beneath the search box. Click the Connect button to connect the repository to Heroku (Fig. F6.3.13). +Once the Heroku App has been created, click the Deploy tab and choose GitHub as the deployment method. Connect to your GitHub account and enter earthengine-apps in the search box. The repository should be listed beneath the search box. Click the Connect button to connect the repository to Heroku (Fig. F6.3.13). -![](F6/image27.png) +![Fig. F6.3.13 Connecting a GitHub account to Heroku](F6/image27.png) -Fig. F6.3.13 Connecting a GitHub account to Heroku -Under the same Deploy tab, scroll down and click Enable Automatic Deploys (Fig. F6.3.14). This will enable Heroku to deploy a new version of the App whenever the GitHub repository gets updated. +Under the same Deploy tab, scroll down and click Enable Automatic Deploys (Fig. F6.3.14). This will enable Heroku to deploy a new version of the App whenever the GitHub repository gets updated. -![](F6/image31.png) +![Fig. F6.3.14 Enabling Automatic Deploys on Heroku](F6/image31.png) -Fig. F6.3.14 Enabling Automatic Deploys on Heroku -Since using Earth Engine requires authentication, we need to set the Earth Engine token as an environment variable so that the web App can pass the Earth Engine authentication. If you have completed Sect. 3, you have successfully authenticated Earth Engine on your computer and the token can be found in the following file path, depending on the operating system you are using. Note that you might need to show the hidden directories on your computer in order to see the .config folder under the home directory. +Since using Earth Engine requires authentication, we need to set the Earth Engine token as an environment variable so that the web App can pass the Earth Engine authentication. If you have completed Sect. 3, you have successfully authenticated Earth Engine on your computer and the token can be found in the following file path, depending on the operating system you are using. Note that you might need to show the hidden directories on your computer in order to see the .config folder under the home directory. Windows: C:UsersUSERNAME.configearthenginecredentialsLinux: /home/USERNAME/.config/earthengine/credentials MacOS: /Users/USERNAME/.config/earthengine/credentials -Open the credentials file using a text editor. Select and copy the long string wrapped by the double quotes (Fig. F6.3.15). Do not include the double quotes in the copied string. +Open the credentials file using a text editor. Select and copy the long string wrapped by the double quotes (Fig. F6.3.15). Do not include the double quotes in the copied string. -![](F6/image68.png) +![Fig. F6.3.15 The Earth Engine authentication token](F6/image68.png) -Fig. F6.3.15 The Earth Engine authentication token -Next, navigate to your web App on Heroku. Click the Settings tab, then click Reveal Config Vars on the page. Enter EARTHENGINE_TOKEN as the key and paste the string copied above as the value. Click the Add button to set EARTHENGINE_TOKEN as an environment variable (Fig. F6.3.16) that will be used by geemap to authenticate Earth Engine. +Next, navigate to your web App on Heroku. Click the Settings tab, then click Reveal Config Vars on the page. Enter EARTHENGINE_TOKEN as the key and paste the string copied above as the value. Click the Add button to set EARTHENGINE_TOKEN as an environment variable (Fig. F6.3.16) that will be used by geemap to authenticate Earth Engine. -![](F6/image4.png) +![Fig. F6.3.16 Setting the Earth Engine token as an environment variable on Heroku](F6/image4.png) -Fig. F6.3.16 Setting the Earth Engine token as an environment variable on Heroku -The last step is to commit some changes to your forked GitHub repository to trigger Heroku to build and deploy the App. If you are familiar with [Git](https://www.google.com/url?q=https://git-scm.com/book/en/v2/Getting-Started-Installing-Git&sa=D&source=editors&ust=1671458841348079&usg=AOvVaw0sqY5Rw_8XGKlIcdI82G_f) commands, you can push changes to the repository from your local computer to GitHub. If you have not used Git before, you can navigate to your repository on GitHub and make changes directly using the browser. For example, you can navigate to [README.md](https://www.google.com/url?q=https://github.com/giswqs/earthengine-apps/blob/master/README.md&sa=D&source=editors&ust=1671458841348610&usg=AOvVaw1JFdaBCpAIcitrc-P7DHhR) and click the Edit icon on the page to start editing the file. Simply place the cursor at the end of the file and press Enter to add an empty line to the file, then click the Commit changes button at the bottom of the page to save changes. This should trigger Heroku to build the App. Check the build status under the latest activities of the Overview tab. Once the App has been built and deployed successfully, you can click the Open app button in the top-right corner to launch the web App (Fig. F6.3.17). When the App is open in your browser, click nlcd_app.ipynb to launch the split-panel map for visualizing land cover change. This is the same notebook that we developed in Sect. 3. The App URL should look like this: [https://APP-NAME.herokuapp.com/voila/render/nlcd_app.ipynb](https://www.google.com/url?q=https://app-name.herokuapp.com/voila/render/nlcd_app.ipynb&sa=D&source=editors&ust=1671458841349532&usg=AOvVaw2GAtxe9X-ymmUvBhjV_PBD). +The last step is to commit some changes to your forked GitHub repository to trigger Heroku to build and deploy the App. If you are familiar with [Git](https://www.google.com/url?q=https://git-scm.com/book/en/v2/Getting-Started-Installing-Git&sa=D&source=editors&ust=1671458841348079&usg=AOvVaw0sqY5Rw_8XGKlIcdI82G_f) commands, you can push changes to the repository from your local computer to GitHub. If you have not used Git before, you can navigate to your repository on GitHub and make changes directly using the browser. For example, you can navigate to [README.md](https://www.google.com/url?q=https://github.com/giswqs/earthengine-apps/blob/master/README.md&sa=D&source=editors&ust=1671458841348610&usg=AOvVaw1JFdaBCpAIcitrc-P7DHhR) and click the Edit icon on the page to start editing the file. Simply place the cursor at the end of the file and press Enter to add an empty line to the file, then click the Commit changes button at the bottom of the page to save changes. This should trigger Heroku to build the App. Check the build status under the latest activities of the Overview tab. Once the App has been built and deployed successfully, you can click the Open app button in the top-right corner to launch the web App (Fig. F6.3.17). When the App is open in your browser, click nlcd_app.ipynb to launch the split-panel map for visualizing land cover change. This is the same notebook that we developed in Sect. 3. The App URL should look like this: [https://APP-NAME.herokuapp.com/voila/render/nlcd_app.ipynb](https://www.google.com/url?q=https://app-name.herokuapp.com/voila/render/nlcd_app.ipynb&sa=D&source=editors&ust=1671458841349532&usg=AOvVaw2GAtxe9X-ymmUvBhjV_PBD). Congratulations! You have successfully deployed the Earth Engine App on Heroku. -![](F6/image51.png) +![Fig. F6.3.17 Setting the Earth Engine token as an environment variable on Heroku](F6/image51.png) -Fig. F6.3.17 Setting the Earth Engine token as an environment variable on Heroku -::: {.callout-note} -Code Checkpoint F63d. The book’s repository contains information about what your code should look like at this point. +:::{.callout-note} +Code Checkpoint F63d. The book’s repository contains information about what your code should look like at this point. ::: -Question 1. What are the pros and cons of designing Earth Engine Apps using geemap and ipywidgets, compared to the JavaScript Earth Engine User Interface API? +Question 1. What are the pros and cons of designing Earth Engine Apps using geemap and ipywidgets, compared to the JavaScript Earth Engine User Interface API? -Question 2. What are the pros and cons of deploying Earth Engine Apps on Heroku, compared to a local web server and the Earth Engine Code Editor? +Question 2. What are the pros and cons of deploying Earth Engine Apps on Heroku, compared to a local web server and the Earth Engine Code Editor? ## Synthesis {.unnumbered} -Assignment 1. Replace the NLCD datasets with other multitemporal land cover datasets (e.g., United States Department of Agriculture National Agricultural Statistics Service Cropland Data Layers, visible in the Earth Engine Data Catalog) and modify the web App for visualizing land cover change using the chosen land cover datasets. Deploy the web App using multiple platforms (i.e., JavaScript Code Editor, ngrok, and Heroku). More land cover datasets can be found at  [https://developers.google.com/earth-engine/datasets/tags/landcover](https://www.google.com/url?q=https://developers.google.com/earth-engine/datasets/tags/landcover&sa=D&source=editors&ust=1671458841352828&usg=AOvVaw0rYGxg8JD8U1d92jnx8V1i). +Assignment 1. Replace the NLCD datasets with other multitemporal land cover datasets (e.g., United States Department of Agriculture National Agricultural Statistics Service Cropland Data Layers, visible in the Earth Engine Data Catalog) and modify the web App for visualizing land cover change using the chosen land cover datasets. Deploy the web App using multiple platforms (i.e., JavaScript Code Editor, ngrok, and Heroku). More land cover datasets can be found at [https://developers.google.com/earth-engine/datasets/tags/landcover](https://www.google.com/url?q=https://developers.google.com/earth-engine/datasets/tags/landcover&sa=D&source=editors&ust=1671458841352828&usg=AOvVaw0rYGxg8JD8U1d92jnx8V1i). ## Conclusion {.unnumbered} -In this chapter, you learned how to design Earth Engine Apps using both the Earth Engine User Interface API (JavaScript) and geemap (Python). You also learned how to deploy Earth Engine Apps on multiple platforms, such as the JavaScript Code Editor, a local web server, and Heroku. The skill of designing and deploying interactive Earth Engine Apps is essential for making your research and data products more accessible to the scientific community and the general public. Anyone with the link to your web App can analyze and visualize Earth Engine datasets without needing an Earth Engine account. +In this chapter, you learned how to design Earth Engine Apps using both the Earth Engine User Interface API (JavaScript) and geemap (Python). You also learned how to deploy Earth Engine Apps on multiple platforms, such as the JavaScript Code Editor, a local web server, and Heroku. The skill of designing and deploying interactive Earth Engine Apps is essential for making your research and data products more accessible to the scientific community and the general public. Anyone with the link to your web App can analyze and visualize Earth Engine datasets without needing an Earth Engine account. ## References {.unnumbered} -* Earth Engine User Interface API: [https://developers.google.com/earth-engine/guides/ui](https://www.google.com/url?q=https://developers.google.com/earth-engine/guides/ui&sa=D&source=editors&ust=1671458841355402&usg=AOvVaw1ZgOA5XvSnn5Uo83ccgHPT)  -* Earth Engine Apps: [https://developers.google.com/earth-engine/guides/apps](https://www.google.com/url?q=https://developers.google.com/earth-engine/guides/apps&sa=D&source=editors&ust=1671458841355890&usg=AOvVaw1ZYwRsMtxZ2RPAn-dC-E0n)  +* Earth Engine User Interface API: [https://developers.google.com/earth-engine/guides/ui](https://www.google.com/url?q=https://developers.google.com/earth-engine/guides/ui&sa=D&source=editors&ust=1671458841355402&usg=AOvVaw1ZgOA5XvSnn5Uo83ccgHPT) +* Earth Engine Apps: [https://developers.google.com/earth-engine/guides/apps](https://www.google.com/url?q=https://developers.google.com/earth-engine/guides/apps&sa=D&source=editors&ust=1671458841355890&usg=AOvVaw1ZYwRsMtxZ2RPAn-dC-E0n) * Voilà: [https://voila.readthedocs.io](https://www.google.com/url?q=https://voila.readthedocs.io&sa=D&source=editors&ust=1671458841356320&usg=AOvVaw1_h1OkQnXAvfEh717DURMY) * geemap: [https://geemap.org](https://www.google.com/url?q=https://geemap.org&sa=D&source=editors&ust=1671458841356762&usg=AOvVaw3ednZ3gbVzDuptXKRZgKCT) * ngrok: [https://ngrok.com](https://www.google.com/url?q=https://ngrok.com&sa=D&source=editors&ust=1671458841357236&usg=AOvVaw11-kfA1McHDaqztFMUrmtP) -* Heroku: [https://heroku.com](https://www.google.com/url?q=https://heroku.com&sa=D&source=editors&ust=1671458841357713&usg=AOvVaw15mVX-rHKo54H2oq0Msd2Z)  -* Earthengine-apps: [https://github.com/giswqs/earthengine-apps](https://www.google.com/url?q=https://github.com/giswqs/earthengine-apps&sa=D&source=editors&ust=1671458841358120&usg=AOvVaw2XD4jjXV9SfPmaC-oLbstc)    +* Heroku: [https://heroku.com](https://www.google.com/url?q=https://heroku.com&sa=D&source=editors&ust=1671458841357713&usg=AOvVaw15mVX-rHKo54H2oq0Msd2Z) +* Earthengine-apps: [https://github.com/giswqs/earthengine-apps](https://www.google.com/url?q=https://github.com/giswqs/earthengine-apps&sa=D&source=editors&ust=1671458841358120&usg=AOvVaw2XD4jjXV9SfPmaC-oLbstc) -Chapter F6.4: Combining R and Earth Engine +Chapter F6.4: Combining R and Earth Engine -::: {.callout-tip} +:::{.callout-tip} # Chapter Information ## Author {.unlisted .unnumbered} -  + Cesar Aybar, David Montero, Antony Barja, Fernando Herrera, Andrea Gonzales, and Wendy Espinoza @@ -2203,13 +2225,13 @@ Cesar Aybar, David Montero, Antony Barja, Fernando Herrera, Andrea Gonzales, and ## Overview {.unlisted .unnumbered} -The purpose of this chapter is to introduce rgee, a non-official API for Earth Engine. You will explore the main features available in rgee and how to set up an environment that integrates rgee with third-party R and Python packages. After this chapter, you will be able to combine R, Python, and JavaScript in the same workflow. +The purpose of this chapter is to introduce rgee, a non-official API for Earth Engine. You will explore the main features available in rgee and how to set up an environment that integrates rgee with third-party R and Python packages. After this chapter, you will be able to combine R, Python, and JavaScript in the same workflow. ## Learning Outcomes {.unlisted .unnumbered} * Becoming familiar with rgee, the Earth Engine R API interface. -* Integrating rgee with other R packages. +* Integrating rgee with other R packages. * Displaying interactive maps. * Integrating Python and R packages using reticulate. * Combining Earth Engine JavaScript and Python APIs with R. @@ -2218,18 +2240,18 @@ The purpose of this chapter is to introduce rgee, a non-official API for Earth E * Install the Python environment (Chap. F6.3). -* Use the require function to load code from existing modules (Chap. F6.1). +* Use the require function to load code from existing modules (Chap. F6.1). * Use the basic functions and logic of Python. * Configure an environment variable and use .Renviron files. * Create Python virtual environments. -::: +::: ## Introduction {.unlisted .unnumbered} -R is a popular programming language established in statistical science with large support in reproducible research, geospatial analysis, data visualization, and much more. To get started with R, you will need to satisfy some extra software requirements. First, install an up-to-date R version (at least 4.0) in your work environment. The installation procedure will vary depending on your operating system (i.e., Windows, Mac, or Linux). Hands-On Programming with R (Garrett Grolemund 2014, Appendix A) explains step by step how to proceed. We strongly recommend that Windows users install Rtools to avoid compiling external static libraries. +R is a popular programming language established in statistical science with large support in reproducible research, geospatial analysis, data visualization, and much more. To get started with R, you will need to satisfy some extra software requirements. First, install an up-to-date R version (at least 4.0) in your work environment. The installation procedure will vary depending on your operating system (i.e., Windows, Mac, or Linux). Hands-On Programming with R (Garrett Grolemund 2014, Appendix A) explains step by step how to proceed. We strongly recommend that Windows users install Rtools to avoid compiling external static libraries. -If you are new to R, a good starting point is the book Geocomputation with R (Lovelace et al. 2019) or Spatial Data Science with Application in R (Pebesma and Bivand 2021). In addition, we recommend using an integrated development environment (e.g., Rstudio) or a code editor (e.g., Visual Studio Code) to create a suitable setting to display and interact with R objects. +If you are new to R, a good starting point is the book Geocomputation with R (Lovelace et al. 2019) or Spatial Data Science with Application in R (Pebesma and Bivand 2021). In addition, we recommend using an integrated development environment (e.g., Rstudio) or a code editor (e.g., Visual Studio Code) to create a suitable setting to display and interact with R objects. The following R packages must be installed (find more information in the R manual) in order to go through the practicum section. @@ -2247,21 +2269,20 @@ install.packages('magick') # Advanced Graphics and Image-Processing in R install.packages('leaflet.extras2') # Extra Functionality for leaflet install.packages('cptcity') # colour gradients from the 'cpt-city' web archive -Earth Engine officially supports client libraries only for the JavaScript and Python programming languages. While the Earth Engine Code Editor offers a convenient environment for rapid prototyping in JavaScript, the lack of a mechanism for integration with local environments makes the development of complex scripts tricky. On the other hand, the Python client library offers much versatility, enabling support for third-party packages. However, not all Earth and environmental scientists code in Python. Hence, a significant number of professionals are not members of the Earth Engine community. In the R ecosystem, rgee (Aybar et al. 2020) tries to fill this gap by wrapping the Earth Engine Python API via reticulate (Kevin Ushey et al. 2021). The rgee package extends and supports all the Earth Engine classes, modules, and functions, working as fast as the other APIs. +Earth Engine officially supports client libraries only for the JavaScript and Python programming languages. While the Earth Engine Code Editor offers a convenient environment for rapid prototyping in JavaScript, the lack of a mechanism for integration with local environments makes the development of complex scripts tricky. On the other hand, the Python client library offers much versatility, enabling support for third-party packages. However, not all Earth and environmental scientists code in Python. Hence, a significant number of professionals are not members of the Earth Engine community. In the R ecosystem, rgee (Aybar et al. 2020) tries to fill this gap by wrapping the Earth Engine Python API via reticulate (Kevin Ushey et al. 2021). The rgee package extends and supports all the Earth Engine classes, modules, and functions, working as fast as the other APIs. -![](F6/image48.png) - -Fig. F6.4.1 A simplified diagram of rgee functionality - -Figure F6.4.1 illustrates how rgee bridges the Earth Engine platform with the R ecosystem. When an Earth Engine request is created in R, rgee transforms this piece of code into Python. Next, the Earth Engine Python API converts the Python code into JSON. Finally, the JSON file request is received by the server through a Web REST API. Users could get the response using the getInfo method by following the same path in reverse. +![Fig. F6.4.1 A simplified diagram of rgee functionality](F6/image48.png) -## Installing rgee  +Figure F6.4.1 illustrates how rgee bridges the Earth Engine platform with the R ecosystem. When an Earth Engine request is created in R, rgee transforms this piece of code into Python. Next, the Earth Engine Python API converts the Python code into JSON. Finally, the JSON file request is received by the server through a Web REST API. Users could get the response using the getInfo method by following the same path in reverse. -To run, rgee needs a Python environment with two packages: NumPy and earthengine-api. Because instructions change frequently, installation is explained at the following checkpoint: -::: {.callout-note} -Code Checkpoint F64a. The book’s repository contains information about setting up the rgee environment. +## Installing rgee + +To run, rgee needs a Python environment with two packages: NumPy and earthengine-api. Because instructions change frequently, installation is explained at the following checkpoint: + +:::{.callout-note} +Code Checkpoint F64a. The book’s repository contains information about setting up the rgee environment. ::: After installing both the R and Python requirements, you can now initialize your Earth Engine account from within R. Consider that R, in contrast to Javascript and Python, supports three distinct Google APIs: Earth Engine, Google Drive, and Google Cloud Storage (GCS). @@ -2272,96 +2293,95 @@ ee_Initialize() # Initialize Earth Engine and GDee_Initialize(drive = TRUE) -If the Google account is verified and the permission is granted, you will be directed to an authentication token. Copy and paste this token into your R console. Consider that the GCS API requires setting up credentials manually; look up and explore the rgee vignette for more information. The verification step is only required once; after that, rgee saves the credentials in your system. +If the Google account is verified and the permission is granted, you will be directed to an authentication token. Copy and paste this token into your R console. Consider that the GCS API requires setting up credentials manually; look up and explore the rgee vignette for more information. The verification step is only required once; after that, rgee saves the credentials in your system. -::: {.callout-note} -Code Checkpoint F64b. The book’s repository contains information about what your code should look like at this point. +:::{.callout-note} +Code Checkpoint F64b. The book’s repository contains information about what your code should look like at this point. ::: -## Creating a 3D Population Density Map with rgee and rayshader +## Creating a 3D Population Density Map with rgee and rayshader -First, import the rgee, rayshader, and raster packages. +First, import the rgee, rayshader, and raster packages. library(rayshader) library(raster) library(rgee) -Initialize the Earth Engine and Google Drive APIs using ee_Initialize. Both credentials must come from the same Google account. +Initialize the Earth Engine and Google Drive APIs using ee_Initialize. Both credentials must come from the same Google account. ee_Initialize(drive = TRUE) -Then, we will access the WorldPop Global Project Population Data dataset. In rgee, the Earth Engine spatial classes (ee$Image, ee$ImageCollection, and ee$FeatureCollection) have a special attribute called Dataset. Users can use it along with autocompletion to quickly find the desired dataset. +Then, we will access the WorldPop Global Project Population Data dataset. In rgee, the Earth Engine spatial classes (ee$Image, ee$ImageCollection, and ee$FeatureCollection) have a special attribute called Dataset. Users can use it along with autocompletion to quickly find the desired dataset. collections <- ee$ImageCollection$Dataset population_data <- collections$CIESIN_GPWv411_GPW_Population_Density population_data_max <- population_data$max() -If you need more information about the Dataset, use ee_utils_dataset_display to go to the official documentation in the Earth Engine Data Catalog. +If you need more information about the Dataset, use ee_utils_dataset_display to go to the official documentation in the Earth Engine Data Catalog. population_data %>% ee_utils_dataset_display() -The rgee package provides various built-in functions to retrieve data from Earth Engine (Aybar et al. 2020). In this example, we use ee_as_raster, which automatically converts an ee$Image (server object) into a RasterLayer (local object). +The rgee package provides various built-in functions to retrieve data from Earth Engine (Aybar et al. 2020). In this example, we use ee_as_raster, which automatically converts an ee$Image (server object) into a RasterLayer (local object). sa_extent <- ee$Geometry$Rectangle( - coords = c(-100, -50, -20, 12), - geodesic = TRUE, - proj = "EPSG:4326") + coords = c(-100, -50, -20, 12), + geodesic = TRUE, + proj = "EPSG:4326") population_data_ly_local <- ee_as_raster( - image = population_data_max, - region = sa_extent, - dsn = "/home/pc-user01/population.tif", # change for your own path. scale = 5000 + image = population_data_max, + region = sa_extent, + dsn = "/home/pc-user01/population.tif", # change for your own path. scale = 5000 ) -Now, turn a RasterLayer into a matrix suitable for rayshader. +Now, turn a RasterLayer into a matrix suitable for rayshader. pop_matrix <- raster_to_matrix(population_data_ly_local) Next, modify the matrix population density values, adding: -* Texture, based on five colors (lightcolor, shadowcolor, leftcolor, rightcolor, and centercolor; see rayshader::create_texture documentation) +* Texture, based on five colors (lightcolor, shadowcolor, leftcolor, rightcolor, and centercolor; see rayshader::create_texture documentation) * Color and shadows (rayshader::sphere_shade) pop_matrix %>% - sphere_shade( -   texture = create_texture("#FFFFFF", "#0800F0", "#FFFFFF", "#FFFFFF", "#FFFFFF") - ) %>% - plot_3d( -   pop_matrix, -   zoom = 0.55, theta = 0, zscale = 100, soliddepth = -24, -   solidcolor = "#525252", shadowdepth = -40, shadowcolor = "black", -   shadowwidth = 25, windowsize = c(800, 720) - ) + sphere_shade( + texture = create_texture("#FFFFFF", "#0800F0", "#FFFFFF", "#FFFFFF", "#FFFFFF") + ) %>% + plot_3d( + pop_matrix, + zoom = 0.55, theta = 0, zscale = 100, soliddepth = -24, + solidcolor = "#525252", shadowdepth = -40, shadowcolor = "black", + shadowwidth = 25, windowsize = c(800, 720) + ) -Lastly, define a title and subtitle for the plot. Use rayshader::render_snapshot to export the final results (Fig. F6.4.2). +Lastly, define a title and subtitle for the plot. Use rayshader::render_snapshot to export the final results (Fig. F6.4.2). -text <- paste0( "South Americanpopulation density", - strrep("n", 27), "Source:GPWv411: Population Density (Gridded Population of the World Version 4.11)") +text <- paste0( "South Americanpopulation density", + strrep("n", 27), "Source:GPWv411: Population Density (Gridded Population of the World Version 4.11)") render_snapshot( - filename = "30_poblacionsudamerica.png", - title_text = text, - title_size = 20, - title_color = "black", - title_font = "Roboto bold", - clear = TRUE + filename = "30_poblacionsudamerica.png", + title_text = text, + title_size = 20, + title_color = "black", + title_font = "Roboto bold", + clear = TRUE ) -![](F6/image2.png) +![Fig. F6.4.2 3D population density map of South America](F6/image2.png) -Fig. F6.4.2 3D population density map of South America -::: {.callout-note} -Code Checkpoint F64c. The book’s repository contains information about what your code should look like at this point. +:::{.callout-note} +Code Checkpoint F64c. The book’s repository contains information about what your code should look like at this point. ::: ## Displaying Maps Interactively -Similar to the Code Editor, rgee supports the interactive visualization of spatial Earth Engine objects by Map$addLayer. First, let’s import the rgee and cptcity packages. The cptcity R package is a wrapper to the cpt-city color gradients web archive. +Similar to the Code Editor, rgee supports the interactive visualization of spatial Earth Engine objects by Map$addLayer. First, let’s import the rgee and cptcity packages. The cptcity R package is a wrapper to the cpt-city color gradients web archive. library(rgee) library(cptcity) ee_Initialize() -We’ll select an ee$Image; in this case, the Shuttle Radar Topography Mission 90 m (SRTM-90) Version 4. +We’ll select an ee$Image; in this case, the Shuttle Radar Topography Mission 90 m (SRTM-90) Version 4. dem <- ee$Image$Dataset$CGIAR_SRTM90_V4 @@ -2372,58 +2392,55 @@ Then, we’ll set the visualization parameters as a list with the following elem * palette: a list of CSS-style color strings viz <- list( - min = 600, - max = 6000, - palette = cpt(pal = 'grass_elevation', rev = TRUE) + min = 600, + max = 6000, + palette = cpt(pal = 'grass_elevation', rev = TRUE) ) Then, we’ll create a simple display using Map$addLayer. m1 <- Map$addLayer(dem, visParams = viz, name = "SRTM", shown = TRUE) -Optionally, you could add a custom legend using Map$addLayer (Fig. F6.4.3). +Optionally, you could add a custom legend using Map$addLayer (Fig. F6.4.3). pal <- Map$addLegend(viz) m1 + pal -![](F6/image28.png) +![Fig. F6.4.3 Interactive visualization of SRTM-90 Version 4 elevation values](F6/image28.png) -Fig. F6.4.3 Interactive visualization of SRTM-90 Version 4 elevation values -The procedure to display ee$Geometry, ee$Feature, and ee$FeatureCollections objects is similar to the previous example effected on an ee$Image. Users just need to change the arguments: eeObject and visParams. +The procedure to display ee$Geometry, ee$Feature, and ee$FeatureCollections objects is similar to the previous example effected on an ee$Image. Users just need to change the arguments: eeObject and visParams. First, Earth Engine geometries (Fig. F6.4.4). vector <- ee$Geometry$Point(-77.011,-11.98) %>% - ee$Feature$buffer(50*1000) + ee$Feature$buffer(50*1000) Map$centerObject(vector) Map$addLayer(vector) # eeObject is a ee$Geometry$Polygon. -![](F6/image46.png) +![Fig. F6.4.4 A polygon buffer surrounding the city of Lima, Peru](F6/image46.png) -Fig. F6.4.4 A polygon buffer surrounding the city of Lima, Peru Next, Earth Engine feature collections (Fig. F6.4.5). building <- ee$FeatureCollection$Dataset$ - `GOOGLE_Research_open-buildings_v1_polygon` + `GOOGLE_Research_open-buildings_v1_polygon` Map$setCenter(3.389, 6.492, 17) Map$addLayer(building) # eeObject is a ee$FeatureCollection -![](F6/image23.png) +![Fig. F6.4.5 Building footprints in Lagos, Nigeria](F6/image23.png) -Fig. F6.4.5 Building footprints in Lagos, Nigeria -The rgee functionality also supports the display of ee$ImageCollection via Map$addLayers (note the extra “s” at the end). Map$addLayers will use the same visualization parameters for all the images (Fig. F6.4.6). If you need different visualization parameters per image, use a Map$addLayer within a for loop. +The rgee functionality also supports the display of ee$ImageCollection via Map$addLayers (note the extra “s” at the end). Map$addLayers will use the same visualization parameters for all the images (Fig. F6.4.6). If you need different visualization parameters per image, use a Map$addLayer within a for loop. # Define a ImageCollection etp <- ee$ImageCollection$Dataset$MODIS_NTSG_MOD16A2_105 %>% - ee$ImageCollection$select("ET") %>% - ee$ImageCollection$filterDate('2014-04-01', '2014-06-01') + ee$ImageCollection$select("ET") %>% + ee$ImageCollection$filterDate('2014-04-01', '2014-06-01') # Set viz paramsviz <- list( - min = 0,  max = 300, - palette = cpt(pal = "grass_bcyr", rev = TRUE) + min = 0, max = 300, + palette = cpt(pal = "grass_bcyr", rev = TRUE) ) # Print map results interactively @@ -2431,11 +2448,10 @@ Map$setCenter(0, 0, 1) etpmap <- Map$addLayers(etp, visParams = viz) etpmap -![](F6/image21.png) +![Fig. F6.4.6 MOD16A2 total evapotranspiration values (kg/m^2/8day)](F6/image21.png) -Fig. F6.4.6 MOD16A2 total evapotranspiration values (kg/m^2/8day) -Another useful rgee feature is the comparison operator (|), which creates a slider in the middle of the canvas, permitting quick comparison of two maps. For instance, load a Landsat 4 image: +Another useful rgee feature is the comparison operator (|), which creates a slider in the middle of the canvas, permitting quick comparison of two maps. For instance, load a Landsat 4 image: landsat <- ee$Image('LANDSAT/LT04/C01/T1/LT04_008067_19890917') @@ -2443,43 +2459,42 @@ Calculate the Normalized Difference Snow Index. ndsi <- landsat$normalizedDifference(c('B3', 'B5')) -Define a constant value and use ee$Image$gte to return a binary image where pixels greater than or equal to that value are set as 1 and the rest are set as 0. Next, filter 0 values using ee$Image$updateMask. +Define a constant value and use ee$Image$gte to return a binary image where pixels greater than or equal to that value are set as 1 and the rest are set as 0. Next, filter 0 values using ee$Image$updateMask. ndsiMasked <- ndsi$updateMask(ndsi$gte(0.4)) Define the visualization parameters. vizParams <- list( - bands <- c('B5', 'B4', 'B3'), # vector of three bands (R, G, B). min = 40, - max = 240, - gamma = c(0.95, 1.1, 1) # Gamma correction factor.) + bands <- c('B5', 'B4', 'B3'), # vector of three bands (R, G, B). min = 40, + max = 240, + gamma = c(0.95, 1.1, 1) # Gamma correction factor.) ndsiViz <- list( - min = 0.5, - max = 1, - palette = c('00FFFF', '0000FF') + min = 0.5, + max = 1, + palette = c('00FFFF', '0000FF') ) Center the map on the Huayhuash mountain range in Peru. Map$setCenter(lon = -77.20, lat = -9.85, zoom = 10) -Finally, visualize both maps using the | operator (Fig. F6.4.7). +Finally, visualize both maps using the | operator (Fig. F6.4.7). m2 <- Map$addLayer(ndsiMasked, ndsiViz, 'NDSI masked') m1 <- Map$addLayer(landsat, vizParams, 'false color composite') -m2  | m1 +m2 | m1 -![](F6/image35.png) +![Fig. F6.4.7 False-color composite over the Huayhuash mountain range, Peru](F6/image35.png) -Fig. F6.4.7 False-color composite over the Huayhuash mountain range, Peru -::: {.callout-note} -Code Checkpoint F64d. The book’s repository contains information about what your code should look like at this point. +:::{.callout-note} +Code Checkpoint F64d. The book’s repository contains information about what your code should look like at this point. ::: -## Integrating rgee with Other Python Packages +## Integrating rgee with Other Python Packages -As noted in Sect. 1, rgee set up a Python environment with NumPy and earthengine-api in your system. However, there is no need to limit it to just two Python packages. In this section, you will learn how to use the Python package ndvi2gif to perform a Normalized Difference Vegetation Index (NDVI) multi-seasonal analysis in the Ocoña Valley without leaving R. +As noted in Sect. 1, rgee set up a Python environment with NumPy and earthengine-api in your system. However, there is no need to limit it to just two Python packages. In this section, you will learn how to use the Python package ndvi2gif to perform a Normalized Difference Vegetation Index (NDVI) multi-seasonal analysis in the Ocoña Valley without leaving R. Whenever you want to install a Python package, you must run the following. @@ -2487,52 +2502,51 @@ library(rgee) library(reticulate) ee_Initialize() -The ee_Initialize function not only authenticates your Earth Engine account but also helps reticulate to set up a Python environment compatible with rgee. After running ee_Initialize, use reticulate::install_python to install the desired Python package. +The ee_Initialize function not only authenticates your Earth Engine account but also helps reticulate to set up a Python environment compatible with rgee. After running ee_Initialize, use reticulate::install_python to install the desired Python package. py_install("ndvi2gif") -The previous procedure is needed just once for each Python environment. Once installed, we simply load the package using reticulate::import. +The previous procedure is needed just once for each Python environment. Once installed, we simply load the package using reticulate::import. ngif <- import("ndvi2gif") -Then, we define our study area using ee$Geometry$Rectangle (Fig. F6.4.8), and use the leaflet layers control to switch between basemaps. +Then, we define our study area using ee$Geometry$Rectangle (Fig. F6.4.8), and use the leaflet layers control to switch between basemaps. colca <- c(-73.15315, -16.46289, -73.07465, -16.37857) roi <- ee$Geometry$Rectangle(colca) Map$centerObject(roi) Map$addLayer(roi) -![](F6/image59.png) +![Fig. F6.4.8 A rectangle drawn over the Ocoña Valley, Peru](F6/image59.png) -Fig. F6.4.8 A rectangle drawn over the Ocoña Valley, Peru -In ndvi2gif, there is just one class: NdviSeasonality. It has the following four public methods.  +In ndvi2gif, there is just one class: NdviSeasonality. It has the following four public methods. -* get_export: Exports NDVI year composites in .GeoTIFF format to your local folder. -* get_export_single: Exports single composite as .GeoTIFF to your local folder. -* get_year_composite: Returns the NDVI composites for each year. -* get_gif: Exports NDVI year composites as a .gif to your local folder. +* get_export: Exports NDVI year composites in .GeoTIFF format to your local folder. +* get_export_single: Exports single composite as .GeoTIFF to your local folder. +* get_year_composite: Returns the NDVI composites for each year. +* get_gif: Exports NDVI year composites as a .gif to your local folder. -To run, the NdviSeasonality constructor needs to define the following arguments. +To run, the NdviSeasonality constructor needs to define the following arguments. -* roi: the region of interest -* start_year: the initial year to start to create yearly composites -* end_year: the end year to look for -* sat: the satellite sensor -* key: the aggregation rule that will be used to generate the yearly composite +* roi: the region of interest +* start_year: the initial year to start to create yearly composites +* end_year: the end year to look for +* sat: the satellite sensor +* key: the aggregation rule that will be used to generate the yearly composite -For each year, the get_year_composite method generates an NDVI ee$Image with four bands, one band per season. Color combination between images and bands will allow you to interpret the vegetation phenology over the seasons and years. In ndvi2gif, the seasons are defined as follows. +For each year, the get_year_composite method generates an NDVI ee$Image with four bands, one band per season. Color combination between images and bands will allow you to interpret the vegetation phenology over the seasons and years. In ndvi2gif, the seasons are defined as follows. -* winter = c('-01-01', '-03-31') +* winter = c('-01-01', '-03-31') * spring = c('-04-01', '-06-30') * summer = c('-07-01', '-09-30') * autumn = c('-10-01', '-12-31') myclass <- ngif$NdviSeasonality( - roi = roi, - start_year = 2016L, - end_year = 2020L, - sat = 'Sentinel', # 'Sentinel', 'Landsat', 'MODIS', 'sar' key = 'max' # 'max', 'median', 'perc_90' + roi = roi, + start_year = 2016L, + end_year = 2020L, + sat = 'Sentinel', # 'Sentinel', 'Landsat', 'MODIS', 'sar' key = 'max' # 'max', 'median', 'perc_90' ) # Estimate the median of the yearly composites from 2016 to 2020. @@ -2541,37 +2555,36 @@ median <- myclass$get_year_composite()$median() # Estimate the median of the winter season composites from 2016 to 2020. wintermax <- myclass$get_year_composite()$select('winter')$max() -We can display maps interactively using the Map$addLayer (Fig. F6.4.9), and use the leaflet layers control to switch between basemaps. +We can display maps interactively using the Map$addLayer (Fig. F6.4.9), and use the leaflet layers control to switch between basemaps. Map$addLayer(wintermax, list(min = 0.1, max = 0.8), 'winterMax') | Map$addLayer(median, list(min = 0.1, max = 0.8), 'median') -![](F6/image24.png) +![Fig. F6.4.9 Comparison between the maximum historic winter NDVI and the mean historic NDVI. Colors represent the season when the maximum value occurred.](F6/image24.png) -Fig. F6.4.9 Comparison between the maximum historic winter NDVI and the mean historic NDVI. Colors represent the season when the maximum value occurred. And we can export the results to a GIF format. myclass$get_gif() -To get more information about the ndvi2gif package, visit its [GitHub](https://www.google.com/url?q=https://github.com/Digdgeo/Ndvi2Gif&sa=D&source=editors&ust=1671458841429620&usg=AOvVaw0cUiJ9wlrQB8a2-Whxs0zA) repository. +To get more information about the ndvi2gif package, visit its [GitHub](https://www.google.com/url?q=https://github.com/Digdgeo/Ndvi2Gif&sa=D&source=editors&ust=1671458841429620&usg=AOvVaw0cUiJ9wlrQB8a2-Whxs0zA) repository. -::: {.callout-note} -Code Checkpoint F64e. The book’s repository contains information about what your code should look like at this point. +:::{.callout-note} +Code Checkpoint F64e. The book’s repository contains information about what your code should look like at this point. ::: -## Converting JavaScript Modules to R +## Converting JavaScript Modules to R -In recent years, the Earth Engine community has developed a lot of valuable third-party modules. Some incredible ones are geeSharp (Zuspan 2020), ee-palettes (Donchyts et al. 2020), spectral (Montero 2021), and LandsatLST (Ermida et al. 2020). While some of these modules have been implemented in Python and JavaScript (e.g., geeSharp and spectral), most are available only for JavaScript. This is a critical drawback, because it divides the Earth Engine community by programming languages. For example, if an R user wants to use tagee (Safanelli et al. 2020), the user will have to first translate the entire module to R. +In recent years, the Earth Engine community has developed a lot of valuable third-party modules. Some incredible ones are geeSharp (Zuspan 2020), ee-palettes (Donchyts et al. 2020), spectral (Montero 2021), and LandsatLST (Ermida et al. 2020). While some of these modules have been implemented in Python and JavaScript (e.g., geeSharp and spectral), most are available only for JavaScript. This is a critical drawback, because it divides the Earth Engine community by programming languages. For example, if an R user wants to use tagee (Safanelli et al. 2020), the user will have to first translate the entire module to R. -In order to close this breach, the ee_extra Python package has been developed to unify the Earth Engine community. The philosophy behind ee_extra is that all of its extended functions, classes, and methods must be functional for the JavaScript, Julia, R, and Python client libraries. Currently, ee_extra is the base of the rgeeExtra (Aybar et al. 2021) and eemont (Montero 2021) packages. +In order to close this breach, the ee_extra Python package has been developed to unify the Earth Engine community. The philosophy behind ee_extra is that all of its extended functions, classes, and methods must be functional for the JavaScript, Julia, R, and Python client libraries. Currently, ee_extra is the base of the rgeeExtra (Aybar et al. 2021) and eemont (Montero 2021) packages. To demonstrate the potential of ee_extra, let’s study an example from the Landsat Land Surface Temperature (LST) JavaScript module. The Landsat LST module computes the land surface temperature for Landsat products (Ermida et al. 2020). First we will run it in the Earth Engine Code Editor; then we will replicate those results in R. -First, JavaScript. In a new script in the Code Editor, we must require the Landsat LST module. +First, JavaScript. In a new script in the Code Editor, we must require the Landsat LST module. -var LandsatLST = require(   'users/sofiaermida/landsat_smw_lst:modules/Landsat_LST.js'); +var LandsatLST = require( 'users/sofiaermida/landsat_smw_lst:modules/Landsat_LST.js'); -The Landsat LST module contains a function named collection. This function receives the following parameters. +The Landsat LST module contains a function named collection. This function receives the following parameters. * The Landsat archive ID * The starting date of the Landsat collection @@ -2581,49 +2594,48 @@ The Landsat LST module contains a function named collection. This function rece In the following code block, we are going to define all required parameters. -var geometry = ee.Geometry.Rectangle([-8.91, 40.0, -8.3, 40.4]); -var satellite = 'L8'; -var date_start = '2018-05-15'; -var date_end = '2018-05-31'; -var use_ndvi = true; +var geometry = ee.Geometry.Rectangle([-8.91, 40.0, -8.3, 40.4]); +var satellite = 'L8'; +var date_start = '2018-05-15'; +var date_end = '2018-05-31'; +var use_ndvi = true; Now, with all our parameters defined, we can compute the land surface temperature by using the collection method from Landsat LST. -var LandsatColl = LandsatLST.collection(satellite, date_start, -   date_end, geometry, use_ndvi); +var LandsatColl = LandsatLST.collection(satellite, date_start, + date_end, geometry, use_ndvi); -The result is stored as an ImageCollection in the LandsatColl variable. Now select the first element of the collection as an example by using the first method. +The result is stored as an ImageCollection in the LandsatColl variable. Now select the first element of the collection as an example by using the first method. -var exImage = LandsatColl.first(); +var exImage = LandsatColl.first(); This example image is now stored in a variable named 'exImage'. Let’s display the LST result on the Map. For visualization purposes, we’ll define a color palette. -var cmap = ['blue', 'cyan', 'green', 'yellow', 'red']; +var cmap = ['blue', 'cyan', 'green', 'yellow', 'red']; Then, we’ll center the map in the region of interest. Map.centerObject(geometry); -Finally, let’s display the LST with the cmap color palette by using the Map.addLayer method (Fig. F6.4.10). This method receives the image to visualize, the visualization parameters, the color palette, and the name of the layer to show in the layer control. The visualization parameters will be: +Finally, let’s display the LST with the cmap color palette by using the Map.addLayer method (Fig. F6.4.10). This method receives the image to visualize, the visualization parameters, the color palette, and the name of the layer to show in the layer control. The visualization parameters will be: * min: 290 (a minimum LST value of 290 K) * max: 320 (a maximum LST value of 320 K) -* palette: cmap (the color palette that was created some steps before) +* palette: cmap (the color palette that was created some steps before) The name of the layer in the Map layer set will be LST. Map.addLayer(exImage.select('LST'), { -   min: 290, -   max: 320, -   palette: cmap + min: 290, + max: 320, + palette: cmap }, 'LST') -![](F6/image45.png) +![Fig. F6.4.10 A map illustrating LST, obtained by following the JavaScript example](F6/image45.png) -Fig. F6.4.10 A map illustrating LST, obtained by following the JavaScript example -::: {.callout-note} -Code Checkpoint F64f. The book’s repository contains a script that shows what your code should look like at this point. +:::{.callout-note} +Code Checkpoint F64f. The book’s repository contains a script that shows what your code should look like at this point. ::: Now, let’s use R to implement the same logic. As in the previous sections, import the R packages: rgee and rgeeExtra. Then, initialize your Earth Engine session. @@ -2637,7 +2649,7 @@ Install rgeeExtra Python dependencies. py_install(packages = c("regex", "ee_extra", "jsbeautifier")) -Using the function rgeeExtra::module loads the JavaScript module. +Using the function rgeeExtra::module loads the JavaScript module. LandsatLST <- module("users/sofiaermida/landsat_smw_lst:modules/Landsat_LST.js") @@ -2654,12 +2666,11 @@ lmod <- list(min = 290, max = 320, palette = cmap) Map$centerObject(geometry) Map$addLayer(exImage$select('LST'), lmod, 'LST') -![](F6/image7.png) +![Fig. F6.4.11 A map illustrating LST, obtained by following the R example](F6/image7.png) -Fig. F6.4.11 A map illustrating LST, obtained by following the R example -::: {.callout-note} -Code Checkpoint F64g. The book’s repository contains information about what your code should look like at this point. +:::{.callout-note} +Code Checkpoint F64g. The book’s repository contains information about what your code should look like at this point. ::: Question 1. When and why might users prefer to use R instead of Python to connect to Earth Engine? @@ -2667,7 +2678,7 @@ Question 2. What are the advantages and disadvantages of using rgee instead of t ## Synthesis {.unnumbered} -Assignment 1. Estimate the Gaussian curvature map from a digital elevation model using rgee and rgeeExtra. Hint: Use the module 'users/joselucassafanelli/TAGEE:TAGEE-functions'. +Assignment 1. Estimate the Gaussian curvature map from a digital elevation model using rgee and rgeeExtra. Hint: Use the module 'users/joselucassafanelli/TAGEE:TAGEE-functions'. ## Conclusion {.unnumbered} diff --git a/docs/F2.html b/docs/F2.html index 2c55a83..095dc72 100644 --- a/docs/F2.html +++ b/docs/F2.html @@ -20,6 +20,69 @@ ul.task-list li input[type="checkbox"] { margin: 0 0.8em 0.2em -1.6em; vertical-align: middle; } +pre > code.sourceCode { white-space: pre; position: relative; } +pre > code.sourceCode > span { display: inline-block; line-height: 1.25; } +pre > code.sourceCode > span:empty { height: 1.2em; } +.sourceCode { overflow: visible; } +code.sourceCode > span { color: inherit; text-decoration: inherit; } +div.sourceCode { margin: 1em 0; } +pre.sourceCode { margin: 0; } +@media screen { +div.sourceCode { overflow: auto; } +} +@media print { +pre > code.sourceCode { white-space: pre-wrap; } +pre > code.sourceCode > span { text-indent: -5em; padding-left: 5em; } +} +pre.numberSource code + { counter-reset: source-line 0; } +pre.numberSource code > span + { position: relative; left: -4em; counter-increment: source-line; } +pre.numberSource code > span > a:first-child::before + { content: counter(source-line); + position: relative; left: -1em; text-align: right; vertical-align: baseline; + border: none; display: inline-block; + -webkit-touch-callout: none; -webkit-user-select: none; + -khtml-user-select: none; -moz-user-select: none; + -ms-user-select: none; user-select: none; + padding: 0 4px; width: 4em; + color: #aaaaaa; + } +pre.numberSource { margin-left: 3em; border-left: 1px solid #aaaaaa; padding-left: 4px; } +div.sourceCode + { } +@media screen { +pre > code.sourceCode > span > a:first-child::before { text-decoration: underline; } +} +code span.al { color: #ff0000; font-weight: bold; } /* Alert */ +code span.an { color: #60a0b0; font-weight: bold; font-style: italic; } /* Annotation */ +code span.at { color: #7d9029; } /* Attribute */ +code span.bn { color: #40a070; } /* BaseN */ +code span.bu { color: #008000; } /* BuiltIn */ +code span.cf { color: #007020; font-weight: bold; } /* ControlFlow */ +code span.ch { color: #4070a0; } /* Char */ +code span.cn { color: #880000; } /* Constant */ +code span.co { color: #60a0b0; font-style: italic; } /* Comment */ +code span.cv { color: #60a0b0; font-weight: bold; font-style: italic; } /* CommentVar */ +code span.do { color: #ba2121; font-style: italic; } /* Documentation */ +code span.dt { color: #902000; } /* DataType */ +code span.dv { color: #40a070; } /* DecVal */ +code span.er { color: #ff0000; font-weight: bold; } /* Error */ +code span.ex { } /* Extension */ +code span.fl { color: #40a070; } /* Float */ +code span.fu { color: #06287e; } /* Function */ +code span.im { color: #008000; font-weight: bold; } /* Import */ +code span.in { color: #60a0b0; font-weight: bold; font-style: italic; } /* Information */ +code span.kw { color: #007020; font-weight: bold; } /* Keyword */ +code span.op { color: #666666; } /* Operator */ +code span.ot { color: #007020; } /* Other */ +code span.pp { color: #bc7a00; } /* Preprocessor */ +code span.sc { color: #4070a0; } /* SpecialChar */ +code span.ss { color: #bb6688; } /* SpecialString */ +code span.st { color: #4070a0; } /* String */ +code span.va { color: #19177c; } /* Variable */ +code span.vs { color: #4070a0; } /* VerbatimString */ +code span.wa { color: #60a0b0; font-weight: bold; font-style: italic; } /* Warning */ @@ -231,14 +294,14 @@ gtag('config', 'G-RK9ZLZQ6GL', { 'anonymize_ip': true}); -
  • 6 Interpreting an Image: Classification +
  • 6 Interpreting an Image: Classification
    • 6.1 Supervised Classification
    • 6.2 Unsupervised Classification
      • @@ -322,96 +385,127 @@ Chapter Information

      Introduction

      -

      Spectral indices are based on the fact that different objects and land covers on the Earth’s surface reflect different amounts of light from the Sun at different wavelengths. In the visible part of the spectrum, for example, a healthy green plant reflects a large amount of green light while absorbing blue and red light—which is why it appears green to our eyes. Light also arrives from the Sun at wavelengths outside what the human eye can see, and there are large differences in reflectances between living and nonliving land covers, and between different types of vegetation, both in the visible and outside the visible wavelengths. We visualized this earlier, in Chaps. F1.1 and F1.3 when we mapped color-infrared images (Fig. F2.0.1).

      -

      -

      Fig. F2.0.1 Mapped color-IR images from multiple satellite sensors that we mapped in Chap. F1.3. The near infrared spectrum is mapped as red, showing where there are high amounts of healthy vegetation.

      +

      Spectral indices are based on the fact that different objects and land covers on the Earth’s surface reflect different amounts of light from the Sun at different wavelengths. In the visible part of the spectrum, for example, a healthy green plant reflects a large amount of green light while absorbing blue and red light—which is why it appears green to our eyes. Light also arrives from the Sun at wavelengths outside what the human eye can see, and there are large differences in reflectances between living and nonliving land covers, and between different types of vegetation, both in the visible and outside the visible wavelengths. We visualized this earlier, in Chaps. F1.1 and F1.3 when we mapped color-infrared images (Fig. F2.0.1).

      +
      +
      +

      +

      Fig. F2.0.1 Mapped color-IR images from multiple satellite sensors that we mapped in Chap. F1.3. The near infrared spectrum is mapped as red, showing where there are high amounts of healthy vegetation.

      +
      +

      If we graph the amount of light (reflectance) at different wavelengths that an object or land cover reflects, we can visualize this more easily (Fig. F2.0.2). For example, look at the reflectance curves for soil and water in the graph below. Soil and water both have relatively low reflectance at wavelengths around 300 nm (ultraviolet and violet light). Conversely, at wavelengths above 700 nm (red and infrared light) soil has relatively high reflectance, while water has very low reflectance. Vegetation, meanwhile, generally reflects large amounts of near infrared light, relative to other land covers.

      -

      -

      Fig. F2.0.2 A graph of the amount of reflectance for different objects on the Earth’s surface at different wavelengths in the visible and infrared portions of the electromagnetic spectrum. 1 micrometer (µm) = 1,000 nanometers (nm).

      -

      Spectral indices use math to express how objects reflect light across multiple portions of the spectrum as a single number. Indices combine multiple bands, often with simple operations of subtraction and division, to create a single value across an image that is intended to help to distinguish particular land uses or land covers of interest. Using Fig. F2.0.2, you can imagine which wavelengths might be the most informative for distinguishing among a variety of land covers. We will explore a variety of calculations made from combinations of bands in the following sections.

      +
      +
      +

      +

      Fig. F2.0.2 A graph of the amount of reflectance for different objects on the Earth’s surface at different wavelengths in the visible and infrared portions of the electromagnetic spectrum. 1 micrometer (µm) = 1,000 nanometers (nm).

      +
      +
      +

      Spectral indices use math to express how objects reflect light across multiple portions of the spectrum as a single number. Indices combine multiple bands, often with simple operations of subtraction and division, to create a single value across an image that is intended to help to distinguish particular land uses or land covers of interest. Using Fig. F2.0.2, you can imagine which wavelengths might be the most informative for distinguishing among a variety of land covers. We will explore a variety of calculations made from combinations of bands in the following sections.

      Indices derived from satellite imagery are used as the basis of many remote-sensing analyses. Indices have been used in thousands of applications, from detecting anthropogenic deforestation to examining crop health. For example, the growth of economically important crops such as wheat and cotton can be monitored throughout the growing season: Bare soil reflects more red wavelengths, whereas growing crops reflect more of the near-infrared (NIR) wavelengths. Thus, calculating a ratio of these two bands can help monitor how well crops are growing (Jackson and Huete 1991).

      5.1 Band Arithmetic in Earth Engine

      -

      If you have not already done so, be sure to add the book’s code repository to the Code Editor by entering https://code.earthengine.google.com/?accept_repo=projects/gee-edu/book into your browser. The book’s scripts will then be available in the script manager panel. If you have trouble finding the repo, you can visit this link for help.

      +

      If you have not already done so, be sure to add the book’s code repository to the Code Editor by entering https://code.earthengine.google.com/?accept_repo=projects/gee-edu/book into your browser. The book’s scripts will then be available in the script manager panel. If you have trouble finding the repo, you can visit this link for help.

      Many indices can be calculated using band arithmetic in Earth Engine. Band arithmetic is the process of adding, subtracting, multiplying, or dividing two or more bands from an image. Here we’ll first do this manually, and then show you some more efficient ways to perform band arithmetic in Earth Engine.

      -

      5.1.1 Arithmetic Calculation of NDVI

      +

      5.1.1 Arithmetic Calculation of NDVI

      The red and near-infrared bands provide a lot of information about vegetation due to vegetation’s high reflectance in these wavelengths. Take a look at Fig. F2.0.2 and note, in particular, that vegetation curves (graphed in green) have relatively high reflectance in the NIR range (approximately 750–900 nm). Also note that vegetation has low reflectance in the red range (approximately 630–690 nm), where sunlight is absorbed by chlorophyll. This suggests that if the red and near-infrared bands could be combined, they would provide substantial information about vegetation.

      Soon after the launch of Landsat 1 in 1972, analysts worked to devise a robust single value that would convey the health of vegetation along a scale of −1 to 1. This yielded the NDVI, using the formula:

      -

                 (F2.0.1)

      -

      where NIR and red refer to the brightness of each of those two bands. As seen in Chaps. F1.1 and F1.2, this brightness might be conveyed in units of reflectance, radiance, or digital number (DN); the NDVI is intended to give nearly equivalent values across platforms that use these wavelengths. The general form of this equation is called a “normalized difference”—the numerator is the “difference” and the denominator “normalizes” the value. Outputs for NDVI vary between −1 and 1. High amounts of green vegetation have values around 0.8–0.9. Absence of green leaves gives values near 0, and water gives values near −1.

      +

      (F2.0.1)

      +

      where NIR and red refer to the brightness of each of those two bands. As seen in Chaps. F1.1 and F1.2, this brightness might be conveyed in units of reflectance, radiance, or digital number (DN); the NDVI is intended to give nearly equivalent values across platforms that use these wavelengths. The general form of this equation is called a “normalized difference”—the numerator is the “difference” and the denominator “normalizes” the value. Outputs for NDVI vary between −1 and 1. High amounts of green vegetation have values around 0.8–0.9. Absence of green leaves gives values near 0, and water gives values near −1.

      To compute the NDVI, we will introduce Earth Engine’s implementation of band arithmetic. Cloud-based band arithmetic is one of the most powerful aspects of Earth Engine, because the platform’s computers are optimized for this type of heavy processing. Arithmetic on bands can be done even at planetary scale very quickly—an idea that was out of reach before the advent of cloud-based remote sensing. Earth Engine automatically partitions calculations across a large number of computers as needed, and assembles the answer for display.

      As an example, let’s examine an image of San Francisco (​​Fig. F2.0.3).

      -

      /////
      -// Band Arithmetic
      -/////

      -

      // Calculate NDVI using Sentinel 2

      -

      // Import and filter imagery by location and date.
      -var sfoPoint = ee.Geometry.Point(-122.3774, 37.6194);
      -var sfoImage = ee.ImageCollection(‘COPERNICUS/S2’)
      -   .filterBounds(sfoPoint)
      -   .filterDate(‘2020-02-01’, ‘2020-04-01’)
      -   .first();

      -

      // Display the image as a false color composite.
      -Map.centerObject(sfoImage, 11);
      -Map.addLayer(sfoImage, {
      -   bands: [‘B8’, ‘B4’, ‘B3’],
      -   min: 0,
      -   max: 2000}, ‘False color’);

      -

      -

      Fig. F2.0.3 False color Sentinel-2 imagery of San Francisco and surroundings

      -

      The simplest mathematical operations in Earth Engine are the add, subtract, multiply, and divide methods. Let’s select the near-infrared and red bands and use these operations to calculate NDVI for our image.

      -

      // Extract the near infrared and red bands.
      -var nir = sfoImage.select(‘B8’);
      -var red = sfoImage.select(‘B4’);

      -

      // Calculate the numerator and the denominator using subtraction and addition respectively.
      -var numerator = nir.subtract(red);
      -var denominator = nir.add(red);

      -

      // Now calculate NDVI.
      -var ndvi = numerator.divide(denominator);

      -

      // Add the layer to our map with a palette.
      -var vegPalette = [‘red’, ‘white’, ‘green’];
      -Map.addLayer(ndvi, {
      -   min: -1,
      -   max: 1,
      -   palette: vegPalette
      -}, ‘NDVI Manual’);

      +
      /////  
+// Band Arithmetic  
+/////  
+  
+// Calculate NDVI using Sentinel 2  
+  
+// Import and filter imagery by location and date.  
+var sfoPoint = ee.Geometry.Point(-122.3774, 37.6194);  
+var sfoImage = ee.ImageCollection('COPERNICUS/S2')  
+   .filterBounds(sfoPoint)  
+   .filterDate('2020-02-01', '2020-04-01')  
+   .first();  
+  
+// Display the image as a false color composite.  
+Map.centerObject(sfoImage, 11);  
+Map.addLayer(sfoImage, {  
+   bands: ['B8', 'B4', 'B3'],  
+   min: 0,  
+   max: 2000}, 'False color');
      +
      +
      +

      +

      Fig. F2.0.3 False color Sentinel-2 imagery of San Francisco and surroundings

      +
      +
      +

      The simplest mathematical operations in Earth Engine are the add, subtract, multiply, and divide methods. Let’s select the near-infrared and red bands and use these operations to calculate NDVI for our image.

      +
      // Extract the near infrared and red bands.  
+var nir = sfoImage.select('B8');  
+var red = sfoImage.select('B4');  
+  
+// Calculate the numerator and the denominator using subtraction and addition respectively.  
+var numerator = nir.subtract(red);  
+var denominator = nir.add(red);  
+  
+// Now calculate NDVI.  
+var ndvi = numerator.divide(denominator);  
+  
+// Add the layer to our map with a palette.  
+var vegPalette = ['red', 'white', 'green'];  
+Map.addLayer(ndvi, {  
+   min: -1,  
+   max: 1,  
+   palette: vegPalette  
+}, 'NDVI Manual');

      Examine the resulting index, using the Inspector to pick out pixel values in areas of vegetation and non-vegetation if desired.

      -

      -

      Fig. F2.0.4 NDVI calculated using Sentinel-2. Remember that outputs for NDVI vary between −1 and 1. High amounts of green vegetation have values around 0.8–0.9. Absence of green leaves gives values near 0, and water gives values near −1.

      +
      +
      +

      +

      Fig. F2.0.4 NDVI calculated using Sentinel-2. Remember that outputs for NDVI vary between −1 and 1. High amounts of green vegetation have values around 0.8–0.9. Absence of green leaves gives values near 0, and water gives values near −1.

      +
      +

      Using these simple arithmetic tools, you can build almost any index, or develop and visualize your own. Earth Engine allows you to quickly and easily calculate and display the index across a large area.

      5.1.2 Single-Operation Computation of Normalized Difference for NDVI

      -

      Normalized differences like NDVI are so common in remote sensing that Earth Engine provides the ability to do that particular sequence of subtraction, addition, and division in a single step, using the normalizedDifference method. This method takes an input image, along with bands you specify, and creates a normalized difference of those two bands. The NDVI computation previously created with band arithmetic can be replaced with one line of code:

      -

      // Now use the built-in normalizedDifference function to achieve the same outcome.
      -var ndviND = sfoImage.normalizedDifference([‘B8’, ‘B4’]);
      -Map.addLayer(ndviND, {
      -   min: -1,
      -   max: 1,
      -   palette: vegPalette
      -}, ‘NDVI normalizedDiff’);

      -

      Note that the order in which you provide the two bands to normalizedDifference is important. We use B8, the near-infrared band, as the first parameter, and the red band B4 as the second. If your two computations of NDVI do not look identical when drawn to the screen, check to make sure that the order you have for the NIR and red bands is correct.

      +

      Normalized differences like NDVI are so common in remote sensing that Earth Engine provides the ability to do that particular sequence of subtraction, addition, and division in a single step, using the normalizedDifference method. This method takes an input image, along with bands you specify, and creates a normalized difference of those two bands. The NDVI computation previously created with band arithmetic can be replaced with one line of code:

      +
      // Now use the built-in normalizedDifference function to achieve the same outcome.  
+var ndviND = sfoImage.normalizedDifference(['B8', 'B4']);  
+Map.addLayer(ndviND, {  
+   min: -1,  
+   max: 1,  
+   palette: vegPalette  
+}, 'NDVI normalizedDiff');
      +

      Note that the order in which you provide the two bands to normalizedDifference is important. We use B8, the near-infrared band, as the first parameter, and the red band B4 as the second. If your two computations of NDVI do not look identical when drawn to the screen, check to make sure that the order you have for the NIR and red bands is correct.

      5.1.3 Using Normalized Difference for NDWI

      -

      As mentioned, the normalized difference approach is used for many different indices. Let’s apply the same normalizedDifference method to another index.

      -

      The Normalized Difference Water Index (NDWI) was developed by Gao (1996) as an index of vegetation water content. The index is sensitive to changes in the liquid content of vegetation canopies. This means that the index can be used, for example, to detect vegetation experiencing drought conditions or differentiate crop irrigation levels. In dry areas, crops that are irrigated can be differentiated from natural vegetation. It is also sometimes called the Normalized Difference Moisture Index (NDMI). NDWI is formulated as follows:

      -

                                      (F2.0.2)

      +

      As mentioned, the normalized difference approach is used for many different indices. Let’s apply the same normalizedDifference method to another index.

      +

      The Normalized Difference Water Index (NDWI) was developed by Gao (1996) as an index of vegetation water content. The index is sensitive to changes in the liquid content of vegetation canopies. This means that the index can be used, for example, to detect vegetation experiencing drought conditions or differentiate crop irrigation levels. In dry areas, crops that are irrigated can be differentiated from natural vegetation. It is also sometimes called the Normalized Difference Moisture Index (NDMI). NDWI is formulated as follows:

      +
      +
      +

      +

      (F2.0.2)

      +
      +

      where NIR is near-infrared, centered near 860 nm (0.86 μm), and SWIR is short-wave infrared, centered near 1,240 nm (1.24 μm).

      -

      Compute and display NDWI in Earth Engine using the normalizedDifference method. Remember that for Sentinel-2, B8 is the NIR band and B11 is the SWIR band (refer to Chaps. F1.1 and F1.3 to find information about imagery bands).

      -

      // Use normalizedDifference to calculate NDWI
      -var ndwi = sfoImage.normalizedDifference([‘B8’, ‘B11’]);
      -var waterPalette = [‘white’, ‘blue’];
      -Map.addLayer(ndwi, {
      -   min: -0.5,
      -   max: 1,
      -   palette: waterPalette
      -}, ‘NDWI’);

      +

      Compute and display NDWI in Earth Engine using the normalizedDifference method. Remember that for Sentinel-2, B8 is the NIR band and B11 is the SWIR band (refer to Chaps. F1.1 and F1.3 to find information about imagery bands).

      +
      // Use normalizedDifference to calculate NDWI  
+var ndwi = sfoImage.normalizedDifference(['B8', 'B11']);  
+var waterPalette = ['white', 'blue'];  
+Map.addLayer(ndwi, {  
+   min: -0.5,  
+   max: 1,  
+   palette: waterPalette  
+}, 'NDWI');

      Examine the areas of the map that NDVI identified as having a lot of vegetation. Notice which are more blue. This is vegetation that has higher water content.

      -

      -

      Fig. F2.0.5 NDWI displayed for Sentinel-2 over San Francisco

      +
      +
      +

      +

      Fig. F2.0.5 NDWI displayed for Sentinel-2 over San Francisco

      +
      +
      @@ -422,122 +516,158 @@ Note
      -

      Code Checkpoint F20a. The book’s repository contains a script that shows what your code should look like at this point. 

      +

      Code Checkpoint F20a. The book’s repository contains a script that shows what your code should look like at this point.

      5.2 Thresholding, Masking, and Remapping Images

      -

      The previous section in this chapter discussed how to use band arithmetic to manipulate images. Those methods created new continuous values by combining bands within an image. This section uses logical operators to categorize band or index values to create a categorized image.

      +

      The previous section in this chapter discussed how to use band arithmetic to manipulate images. Those methods created new continuous values by combining bands within an image. This section uses logical operators to categorize band or index values to create a categorized image.

      5.2.1 Implementing a Threshold

      -

      Implementing a threshold uses a number (the threshold value) and logical operators to help us partition the variability of images into categories. For example, recall our map of NDVI. High amounts of vegetation have NDVI values near 1 and non-vegetated areas are near 0. If we want to see what areas of the map have vegetation, we can use a threshold to generalize the NDVI value in each pixel as being either “no vegetation” or “vegetation”. That is a substantial simplification, to be sure, but can help us to better comprehend the rich variation on the Earth’s surface. This type of categorization may be useful if, for example, we want to look at the proportion of a city that is vegetated. Let’s create a Sentinel-2 map of NDVI near Seattle, Washington, USA. Enter the code below in a new script.

      -

      // Create an NDVI image using Sentinel 2.
      -var seaPoint = ee.Geometry.Point(-122.2040, 47.6221);
      -var seaImage = ee.ImageCollection(‘COPERNICUS/S2’)
      -   .filterBounds(seaPoint)
      -   .filterDate(‘2020-08-15’, ‘2020-10-01’)
      -   .first();

      -

      var seaNDVI = seaImage.normalizedDifference([‘B8’, ‘B4’]);

      -

      // And map it.
      -Map.centerObject(seaPoint, 10);
      -var vegPalette = [‘red’, ‘white’, ‘green’];
      -Map.addLayer(seaNDVI,
      -   {
      -       min: -1,
      -       max: 1,
      -       palette: vegPalette
      -   },   ‘NDVI Seattle’);

      -

      -

      Fig. F2.0.6 NDVI image of Sentinel-2 imagery over Seattle, Washington, USA

      -

      Inspect the image. We can see that vegetated areas are darker green while non-vegetated locations are white and water is pink. If we use the Inspector to query our image, we can see that parks and other forested areas have an NDVI over about 0.5. Thus, it would make sense to define areas with NDVI values greater than 0.5 as forested, and those below that threshold as not forested.

      +

      Implementing a threshold uses a number (the threshold value) and logical operators to help us partition the variability of images into categories. For example, recall our map of NDVI. High amounts of vegetation have NDVI values near 1 and non-vegetated areas are near 0. If we want to see what areas of the map have vegetation, we can use a threshold to generalize the NDVI value in each pixel as being either “no vegetation” or “vegetation”. That is a substantial simplification, to be sure, but can help us to better comprehend the rich variation on the Earth’s surface. This type of categorization may be useful if, for example, we want to look at the proportion of a city that is vegetated. Let’s create a Sentinel-2 map of NDVI near Seattle, Washington, USA. Enter the code below in a new script.

      +
      // Create an NDVI image using Sentinel 2.  
+var seaPoint = ee.Geometry.Point(-122.2040, 47.6221);  
+var seaImage = ee.ImageCollection('COPERNICUS/S2')  
+   .filterBounds(seaPoint)  
+   .filterDate('2020-08-15', '2020-10-01')  
+   .first();  
+  
+var seaNDVI = seaImage.normalizedDifference(['B8', 'B4']);  
+  
+// And map it.  
+Map.centerObject(seaPoint, 10);  
+var vegPalette = ['red', 'white', 'green'];  
+Map.addLayer(seaNDVI,  
+   {  
+       min: -1,  
+       max: 1,  
+       palette: vegPalette  
+   },   'NDVI Seattle');
      +
      +
      +

      +

      Fig. F2.0.6 NDVI image of Sentinel-2 imagery over Seattle, Washington, USA

      +
      +
      +

      Inspect the image. We can see that vegetated areas are darker green while non-vegetated locations are white and water is pink. If we use the Inspector to query our image, we can see that parks and other forested areas have an NDVI over about 0.5. Thus, it would make sense to define areas with NDVI values greater than 0.5 as forested, and those below that threshold as not forested.

      Now let’s define that value as a threshold and use it to threshold our vegetated areas.

      -

      // Implement a threshold.
      -var seaVeg = seaNDVI.gt(0.5);

      -

      // Map the threshold.
      -Map.addLayer(seaVeg,
      -   {
      -       min: 0,
      -       max: 1,
      -       palette: [‘white’, ‘green’]
      -   },   ‘Non-forest vs. Forest’);

      -

      The gt method is from the family of Boolean operators—that is, gt is a function that performs a test in each pixel and returns the value 1 if the test evaluates to true, and 0 otherwise. Here, for every pixel in the image, it tests whether the NDVI value is greater than 0.5. When this condition is met, the layer seaVeg gets the value 1. When the condition is false, it receives the value 0.

      -

      -

      Fig. F2.0.7 Thresholded forest and non-forest image based on NDVI for Seattle, Washington, USA

      -

      Use the Inspector tool to explore this new layer. If you click on a green location, that NDVI should be greater than 0.5. If you click on a white pixel, the NDVI value should be equal to or less than 0.5.

      +
      // Implement a threshold.  
+var seaVeg = seaNDVI.gt(0.5);  
+  
+// Map the threshold.  
+Map.addLayer(seaVeg,  
+   {  
+       min: 0,  
+       max: 1,  
+       palette: ['white', 'green']  
+   },   'Non-forest vs. Forest');
      +

      The gt method is from the family of Boolean operators—that is, gt is a function that performs a test in each pixel and returns the value 1 if the test evaluates to true, and 0 otherwise. Here, for every pixel in the image, it tests whether the NDVI value is greater than 0.5. When this condition is met, the layer seaVeg gets the value 1. When the condition is false, it receives the value 0.

      +
      +
      +

      +

      Fig. F2.0.7 Thresholded forest and non-forest image based on NDVI for Seattle, Washington, USA

      +
      +
      +

      Use the Inspector tool to explore this new layer. If you click on a green location, that NDVI should be greater than 0.5. If you click on a white pixel, the NDVI value should be equal to or less than 0.5.

      Other operators in this Boolean family include less than (lt), less than or equal to (lte), equal to (eq), not equal to (neq), and greater than or equal to (gte) and more.

      -

      5.2.2 Building Complex Categorizations with .where

      -

      A binary map classifying NDVI is very useful. However, there are situations where you may want to split your image into more than two bins. Earth Engine provides a tool, the where method, that conditionally evaluates to true or false within each pixel depending on the outcome of a test. This is analogous to an if statement seen commonly in other languages. However, to perform this logic when programming for Earth Engine, we avoid using the JavaScript if statement. Importantly, JavaScript if commands are not calculated on Google’s servers, and can create serious problems when running your code—in effect, the servers try to ship all of the information to be executed to your own computer’s browser, which is very underequipped for such enormous tasks. Instead, we use the where clause for conditional logic.

      -

      Suppose instead of just splitting the forested areas from the non-forested areas in our NDVI, we want to split the image into likely water, non-forested, and forested areas. We can use where and thresholds of -0.1 and 0.5. We will start by creating an image using ee.Image. We then clip the new image so that it covers the same area as our seaNDVI layer.

      -

      // Implement .where.
      -// Create a starting image with all values = 1.
      -var seaWhere = ee.Image(1)   // Use clip to constrain the size of the new image.   .clip(seaNDVI.geometry());

      -

      // Make all NDVI values less than -0.1 equal 0.
      -seaWhere = seaWhere.where(seaNDVI.lte(-0.1), 0);

      -

      // Make all NDVI values greater than 0.5 equal 2.
      -seaWhere = seaWhere.where(seaNDVI.gte(0.5), 2);

      -

      // Map our layer that has been divided into three classes.
      -Map.addLayer(seaWhere,
      -   {
      -       min: 0,
      -       max: 2,
      -       palette: [‘blue’, ‘white’, ‘green’]
      -   },   ‘Water, Non-forest, Forest’);

      -

      There are a few interesting things to note about this code that you may not have seen before. First, we’re not defining a new variable for each where call. As a result, we can perform many where calls without creating a new variable each time and needing to keep track of them. Second, when we created the starting image, we set the value to 1. This means that we could easily set the bottom and top values with one where clause each. Finally, while we did not do it here, we can combine multiple where clauses using and and or. For example, we could identify pixels with an intermediate level of NDVI using seaNDVI.gte(-0.1).and(seaNDVI.lt(0.5)).

      -

      -

      Fig. F2.0.8 Thresholded water, forest, and non-forest image based on NDVI for Seattle, Washington, USA.

      +

      5.2.2 Building Complex Categorizations with .where

      +

      A binary map classifying NDVI is very useful. However, there are situations where you may want to split your image into more than two bins. Earth Engine provides a tool, the where method, that conditionally evaluates to true or false within each pixel depending on the outcome of a test. This is analogous to an if statement seen commonly in other languages. However, to perform this logic when programming for Earth Engine, we avoid using the JavaScript if statement. Importantly, JavaScript if commands are not calculated on Google’s servers, and can create serious problems when running your code—in effect, the servers try to ship all of the information to be executed to your own computer’s browser, which is very underequipped for such enormous tasks. Instead, we use the where clause for conditional logic.

      +

      Suppose instead of just splitting the forested areas from the non-forested areas in our NDVI, we want to split the image into likely water, non-forested, and forested areas. We can use where and thresholds of -0.1 and 0.5. We will start by creating an image using ee.Image. We then clip the new image so that it covers the same area as our seaNDVI layer.

      +
      // Implement .where.  
+// Create a starting image with all values = 1.  
+var seaWhere = ee.Image(1)   // Use clip to constrain the size of the new image.   .clip(seaNDVI.geometry());  
+  
+// Make all NDVI values less than -0.1 equal 0.  
+seaWhere = seaWhere.where(seaNDVI.lte(-0.1), 0);  
+  
+// Make all NDVI values greater than 0.5 equal 2.  
+seaWhere = seaWhere.where(seaNDVI.gte(0.5), 2);  
+  
+// Map our layer that has been divided into three classes.  
+Map.addLayer(seaWhere,  
+   {  
+       min: 0,  
+       max: 2,  
+       palette: ['blue', 'white', 'green']  
+   },   'Water, Non-forest, Forest');
      +

      There are a few interesting things to note about this code that you may not have seen before. First, we’re not defining a new variable for each where call. As a result, we can perform many where calls without creating a new variable each time and needing to keep track of them. Second, when we created the starting image, we set the value to 1. This means that we could easily set the bottom and top values with one where clause each. Finally, while we did not do it here, we can combine multiple where clauses using and and or. For example, we could identify pixels with an intermediate level of NDVI using seaNDVI.gte(-0.1).and(seaNDVI.lt(0.5)).

      +
      +
      +

      +

      Fig. F2.0.8 Thresholded water, forest, and non-forest image based on NDVI for Seattle, Washington, USA.

      +
      +

      5.2.3 Masking Specific Values in an Image

      Masking an image is a technique that removes specific areas of an image—those covered by the mask—from being displayed or analyzed. Earth Engine allows you to both view the current mask and update the mask.

      -

      // Implement masking.
      -// View the seaVeg layer’s current mask.
      -Map.centerObject(seaPoint, 9);
      -Map.addLayer(seaVeg.mask(), {}, ‘seaVeg Mask’);

      -

      -

      Fig. F2.0.9 The existing mask for the seaVeg layer we created previously

      +
      // Implement masking.  
+// View the seaVeg layer's current mask.  
+Map.centerObject(seaPoint, 9);  
+Map.addLayer(seaVeg.mask(), {}, 'seaVeg Mask');
      +
      +
      +

      +

      Fig. F2.0.9 The existing mask for the seaVeg layer we created previously

      +
      +

      You can use the Inspector to see that the black area is masked and the white area has a constant value of 1. This means that data values are mapped and available for analysis within the white area only.

      -

      Now suppose we only want to display and conduct analyses in the forested areas. Let’s mask out the non-forested areas from our image. First, we create a binary mask using the equals (eq) method.

      -

      // Create a binary mask of non-forest.
      -var vegMask = seaVeg.eq(1);

      -

      In making a mask, you set the values you want to see and analyze to be a number greater than 0. The idea is to set unwanted values to get the value of 0. Pixels that had 0 values become masked out (in practice, they do not appear on the screen at all) once we use the updateMask method to add these values to the existing mask.

      -

      // Update the seaVeg mask with the non-forest mask.
      -var maskedVeg = seaVeg.updateMask(vegMask);

      -

      // Map the updated Veg layer
      -Map.addLayer(maskedVeg,
      -   {
      -       min: 0,
      -       max: 1,
      -       palette: [‘green’]
      -   },   ‘Masked Forest Layer’);

      -

      Turn off all of the other layers. You can see how the maskedVeg layer now has masked out all non-forested areas.

      -

      -

      Fig. F2.0.10 An updated mask now displays only the forested areas. Non-forested areas are masked out and transparent.

      +

      Now suppose we only want to display and conduct analyses in the forested areas. Let’s mask out the non-forested areas from our image. First, we create a binary mask using the equals (eq) method.

      +
      // Create a binary mask of non-forest.  
+var vegMask = seaVeg.eq(1);
      +

      In making a mask, you set the values you want to see and analyze to be a number greater than 0. The idea is to set unwanted values to get the value of 0. Pixels that had 0 values become masked out (in practice, they do not appear on the screen at all) once we use the updateMask method to add these values to the existing mask.

      +
      // Update the seaVeg mask with the non-forest mask.  
+var maskedVeg = seaVeg.updateMask(vegMask);  
+  
+// Map the updated Veg layer  
+Map.addLayer(maskedVeg,  
+   {  
+       min: 0,  
+       max: 1,  
+       palette: ['green']  
+   },   'Masked Forest Layer');
      +

      Turn off all of the other layers. You can see how the maskedVeg layer now has masked out all non-forested areas.

      +
      +
      +

      +

      Fig. F2.0.10 An updated mask now displays only the forested areas. Non-forested areas are masked out and transparent.

      +
      +

      Map the updated mask for the layer and you can see why this is.

      -

      // Map the updated mask
      -Map.addLayer(maskedVeg.mask(), {}, ‘maskedVeg Mask’);

      -

      -

      Fig. F2.0.11 The updated mask. Areas of non-forest are now masked out as well (black areas of the image).

      +
      // Map the updated mask  
+Map.addLayer(maskedVeg.mask(), {}, 'maskedVeg Mask');
      +
      +
      +

      +

      Fig. F2.0.11 The updated mask. Areas of non-forest are now masked out as well (black areas of the image).

      +
      +

      5.2.4 Remapping Values in an Image

      Remapping takes specific values in an image and assigns them a different value. This is particularly useful for categorical datasets, including those you read about in Chap. F1.2 and those we have created earlier in this chapter.

      -

      Let’s use the remap method to change the values for our seaWhere layer. Note that since we’re changing the middle value to be the largest, we’ll need to adjust our palette as well.

      -

      // Implement remapping.
      -// Remap the values from the seaWhere layer.
      -var seaRemap = seaWhere.remap([0, 1, 2], // Existing values.   [9, 11, 10]); // Remapped values.

      -

      Map.addLayer(seaRemap,
      -   {
      -       min: 9,
      -       max: 11,
      -       palette: [‘blue’, ‘green’, ‘white’]
      -   },   ‘Remapped Values’);

      -

      Use the inspector to compare values between our original seaWhere (displayed as Water, Non-Forest, Forest) and the seaRemap, marked as “Remapped Values.” Click on a forested area and you should see that the Remapped Values should be 10, instead of 2 (Fig. F2.0.12).

      -

      -

      Fig. F2.0.12 For forested areas, the remapped layer has a value of 10, compared with the original layer, which has a value of 2. You may have more layers in your Inspector.

      +

      Let’s use the remap method to change the values for our seaWhere layer. Note that since we’re changing the middle value to be the largest, we’ll need to adjust our palette as well.

      +
      // Implement remapping.  
+// Remap the values from the seaWhere layer.  
+var seaRemap = seaWhere.remap([0, 1, 2], // Existing values.   [9, 11, 10]); // Remapped values.  
+  
+Map.addLayer(seaRemap,  
+   {  
+       min: 9,  
+       max: 11,  
+       palette: ['blue', 'green', 'white']  
+   },   'Remapped Values');
      +

      Use the inspector to compare values between our original seaWhere (displayed as Water, Non-Forest, Forest) and the seaRemap, marked as “Remapped Values.” Click on a forested area and you should see that the Remapped Values should be 10, instead of 2 (Fig. F2.0.12).

      +
      +
      +

      +

      Fig. F2.0.12 For forested areas, the remapped layer has a value of 10, compared with the original layer, which has a value of 2. You may have more layers in your Inspector.

      +
      +
      @@ -548,32 +678,32 @@ Note
      -

      Code Checkpoint F20b. The book’s repository contains a script that shows what your code should look like at this point. 

      +

      Code Checkpoint F20b. The book’s repository contains a script that shows what your code should look like at this point.

      Synthesis

      -

      Assignment 1. In addition to vegetation indices and other land cover indices, you can use properties of different soil types to create geological indices. The Clay Minerals Ratio (CMR) is one of these. This index highlights soils containing clay and alunite, which absorb radiation in the SWIR portion (2.0–2.3 μm) of the spectrum.

      +

      Assignment 1. In addition to vegetation indices and other land cover indices, you can use properties of different soil types to create geological indices. The Clay Minerals Ratio (CMR) is one of these. This index highlights soils containing clay and alunite, which absorb radiation in the SWIR portion (2.0–2.3 μm) of the spectrum.

      -

      SWIR 1 should be in the 1.55–1.75 µm range, and SWIR 2 should be in the 2.08–2.35 µm range. Calculate and display CMR at the following point: ee.Geometry.Point(-100.543, 33.456). Don’t forget to use Map.centerObject.

      -

      We’ve selected an area of Texas known for its clay soils. Compare this with an area without clay soils (for example, try an area around Seattle or Tacoma, Washington, USA). Note that this index will also pick up roads and other paved areas.

      +

      SWIR 1 should be in the 1.55–1.75 µm range, and SWIR 2 should be in the 2.08–2.35 µm range. Calculate and display CMR at the following point: ee.Geometry.Point(-100.543, 33.456). Don’t forget to use Map.centerObject.

      +

      We’ve selected an area of Texas known for its clay soils. Compare this with an area without clay soils (for example, try an area around Seattle or Tacoma, Washington, USA). Note that this index will also pick up roads and other paved areas.

      Assignment 2. Calculate the Iron Oxide Ratio, which can be used to detect hydrothermally altered rocks (e.g., from volcanoes) that contain iron-bearing sulfides which have been oxidized (Segal, 1982).

      Here’s the formula:

      -

      Red should be the 0.63–0.69 µm spectral range and Blue the 0.45–0.52 µm. Using Landsat 8, you can also find an interesting area to map by considering where these types of rocks might occur.

      -

      Assignment 3. Calculate the Normalized Difference Built-Up Index (NDBI) for the sfoImage used in this chapter.

      -

      The NDBI was developed by Zha et al. (2003) to aid in differentiating urban areas (e.g., densely clustered buildings and roads) from other land cover types. The index exploits the fact that urban areas, which generally have a great deal of impervious surface cover, reflect SWIR very strongly. If you like, refer back to Fig. F2.0.2.

      +

      Red should be the 0.63–0.69 µm spectral range and Blue the 0.45–0.52 µm. Using Landsat 8, you can also find an interesting area to map by considering where these types of rocks might occur.

      +

      Assignment 3. Calculate the Normalized Difference Built-Up Index (NDBI) for the sfoImage used in this chapter.

      +

      The NDBI was developed by Zha et al. (2003) to aid in differentiating urban areas (e.g., densely clustered buildings and roads) from other land cover types. The index exploits the fact that urban areas, which generally have a great deal of impervious surface cover, reflect SWIR very strongly. If you like, refer back to Fig. F2.0.2.

      The formula is:

      Using what we know about Sentinel-2 bands, compute NDBI and display it.

      -

      Bonus: Note that NDBI is the negative of NDWI computed earlier. We can prove this by using the JavaScript reverse method to reverse the palette used for NDWI in Earth Engine. This method reverses the order of items in the JavaScript list. Create a new palette for NDBI using the reverse method and display the map. As a hint, here is code to use the reverse method.

      -

      var barePalette = waterPalette.reverse();

      +

      Bonus: Note that NDBI is the negative of NDWI computed earlier. We can prove this by using the JavaScript reverse method to reverse the palette used for NDWI in Earth Engine. This method reverses the order of items in the JavaScript list. Create a new palette for NDBI using the reverse method and display the map. As a hint, here is code to use the reverse method.

      +

      var barePalette = waterPalette.reverse();

      Conclusion

      -

      In this chapter, you learned how to select multiple bands from an image and calculate indices. You also learned about thresholding values in an image, slicing them into multiple categories using thresholds. It is also possible to work with one set of class numbers and remap them quickly to another set. Using these techniques, you have some of the basic tools of image manipulation. In subsequent chapters you will encounter more complex and specialized image manipulation techniques, including pixel-based image transformations (Chap. F3.1), neighborhood-based image transformations (Chap. F3.2), and object-based image analysis (Chap. F3.3).

      +

      In this chapter, you learned how to select multiple bands from an image and calculate indices. You also learned about thresholding values in an image, slicing them into multiple categories using thresholds. It is also possible to work with one set of class numbers and remap them quickly to another set. Using these techniques, you have some of the basic tools of image manipulation. In subsequent chapters you will encounter more complex and specialized image manipulation techniques, including pixel-based image transformations (Chap. F3.1), neighborhood-based image transformations (Chap. F3.2), and object-based image analysis (Chap. F3.3).

      References

      @@ -593,7 +723,7 @@ Note
      -

      6 Interpreting an Image: Classification

      +

      6 Interpreting an Image: Classification

      @@ -610,7 +740,7 @@ Chapter Information

      Overview

      -

      Image classification is a fundamental goal of remote sensing. It takes the user from viewing an image to labeling its contents. This chapter introduces readers to the concept of classification and walks users through the many options for image classification in Earth Engine. You will explore the processes of training data collection, classifier selection, classifier training, and image classification.

      +

      Image classification is a fundamental goal of remote sensing. It takes the user from viewing an image to labeling its contents. This chapter introduces readers to the concept of classification and walks users through the many options for image classification in Earth Engine. You will explore the processes of training data collection, classifier selection, classifier training, and image classification.

      Learning Outcomes

      @@ -633,45 +763,56 @@ Chapter Information

      Introduction

      -

      Classification is addressed in a broad range of fields, including mathematics, statistics, data mining, machine learning, and more. For a deeper treatment of classification, interested readers may see some of the following suggestions: Witten et al. (2011), Hastie et al. (2009), Goodfellow et al. (2016), Gareth et al. (2013), Géron (2019), Müller et al. (2016), or Witten et al. (2005). Unlike regression, which predicts continuous variables, classification predicts categorical, or discrete, variables—variables with a finite number of categories (e.g., age range).

      -

      In remote sensing, image classification is an attempt to categorize all pixels in an image into a finite number of labeled land cover and/or land use classes. The resulting classified image is a simplified thematic map derived from the original image (Fig. F2.1.1). Land cover and land use information is essential for many environmental and socioeconomic applications, including natural resource management, urban planning, biodiversity conservation, agricultural monitoring, and carbon accounting.

      -

      -

      Fig. F2.1.1 Image classification concept

      +

      Classification is addressed in a broad range of fields, including mathematics, statistics, data mining, machine learning, and more. For a deeper treatment of classification, interested readers may see some of the following suggestions: Witten et al. (2011), Hastie et al. (2009), Goodfellow et al. (2016), Gareth et al. (2013), Géron (2019), Müller et al. (2016), or Witten et al. (2005). Unlike regression, which predicts continuous variables, classification predicts categorical, or discrete, variables—variables with a finite number of categories (e.g., age range).

      +

      In remote sensing, image classification is an attempt to categorize all pixels in an image into a finite number of labeled land cover and/or land use classes. The resulting classified image is a simplified thematic map derived from the original image (Fig. F2.1.1). Land cover and land use information is essential for many environmental and socioeconomic applications, including natural resource management, urban planning, biodiversity conservation, agricultural monitoring, and carbon accounting.

      +
      +
      +

      +

      Fig. F2.1.1 Image classification concept

      +
      +

      Image classification techniques for generating land cover and land use information have been in use since the 1980s (Li et al. 2014). Here, we will cover the concepts of pixel-based supervised and unsupervised classifications, testing out different classifiers. Chapter F3.3 covers the concept and application of object-based classification.

      -

      It is important to define land use and land cover. Land cover relates to the physical characteristics of the surface: simply put, it documents whether an area of the Earth’s surface is covered by forests, water, impervious surfaces, etc. Land use refers to how this land is being used by people. For example, herbaceous vegetation is considered a land cover but can indicate different land uses: the grass in a pasture is an agricultural land use, whereas the grass in an urban area can be classified as a park.

      +

      It is important to define land use and land cover. Land cover relates to the physical characteristics of the surface: simply put, it documents whether an area of the Earth’s surface is covered by forests, water, impervious surfaces, etc. Land use refers to how this land is being used by people. For example, herbaceous vegetation is considered a land cover but can indicate different land uses: the grass in a pasture is an agricultural land use, whereas the grass in an urban area can be classified as a park.

      6.1 Supervised Classification

      -

      If you have not already done so, be sure to add the book’s code repository to the Code Editor by entering https://code.earthengine.google.com/?accept_repo=projects/gee-edu/book into your browser. The book’s scripts will then be available in the script manager panel. If you have trouble finding the repo, you can visit this link for help.

      -

      Supervised classification uses a training dataset with known labels and representing the spectral characteristics of each land cover class of interest to “supervise” the classification. The overall approach of a supervised classification in Earth Engine is summarized as follows:

      +

      If you have not already done so, be sure to add the book’s code repository to the Code Editor by entering https://code.earthengine.google.com/?accept_repo=projects/gee-edu/book into your browser. The book’s scripts will then be available in the script manager panel. If you have trouble finding the repo, you can visit this link for help.

      +

      Supervised classification uses a training dataset with known labels and representing the spectral characteristics of each land cover class of interest to “supervise” the classification. The overall approach of a supervised classification in Earth Engine is summarized as follows:

      1. Get a scene.
      2. Collect training data.
      3. Select and train a classifier using the training data.
      4. Classify the image using the selected classifier.
      -

      We will begin by creating training data manually, based on a clear Landsat image (Fig. F2.1.2). Copy the code block below to define your Landsat 8 scene variable and add it to the map. We will use a point in Milan, Italy, as the center of the area for our image classification.

      -

      // Create an Earth Engine Point object over Milan.
      -var pt = ee.Geometry.Point([9.453, 45.424]);

      -

      // Filter the Landsat 8 collection and select the least cloudy image.
      -var landsat = ee.ImageCollection(‘LANDSAT/LC08/C02/T1_L2’)
      -   .filterBounds(pt)
      -   .filterDate(‘2019-01-01’, ‘2020-01-01’)
      -   .sort(‘CLOUD_COVER’)
      -   .first();

      -

      // Center the map on that image.
      -Map.centerObject(landsat, 8);

      -

      // Add Landsat image to the map.
      -var visParams = {
      -   bands: [‘SR_B4’, ‘SR_B3’, ‘SR_B2’],
      -   min: 7000,
      -   max: 12000
      -};
      -Map.addLayer(landsat, visParams, ‘Landsat 8 image’);

      -

      -

      Fig. F2.1.2 Landsat image

      +

      We will begin by creating training data manually, based on a clear Landsat image (Fig. F2.1.2). Copy the code block below to define your Landsat 8 scene variable and add it to the map. We will use a point in Milan, Italy, as the center of the area for our image classification.

      +
      // Create an Earth Engine Point object over Milan.  
+var pt = ee.Geometry.Point([9.453, 45.424]);  
+  
+// Filter the Landsat 8 collection and select the least cloudy image.  
+var landsat = ee.ImageCollection('LANDSAT/LC08/C02/T1_L2')  
+   .filterBounds(pt)  
+   .filterDate('2019-01-01', '2020-01-01')  
+   .sort('CLOUD_COVER')  
+   .first();  
+  
+// Center the map on that image.  
+Map.centerObject(landsat, 8);  
+  
+// Add Landsat image to the map.  
+var visParams = {  
+   bands: ['SR_B4', 'SR_B3', 'SR_B2'],  
+   min: 7000,  
+   max: 12000  
+};  
+Map.addLayer(landsat, visParams, 'Landsat 8 image');
      +
      +
      +

      +

      Fig. F2.1.2 Landsat image

      +
      +

      Using the Geometry Tools, we will create points on the Landsat image that represent land cover classes of interest to use as our training data. We’ll need to do two things: (1) identify where each land cover occurs on the ground, and (2) label the points with the proper class number. For this exercise, we will use the classes and codes shown in Table 2.1.1.

      -

      Table 2.1.1 Land cover classes

      +

      Table 2.1.1 Land cover classes

      Class

      Class code

      Forest

      @@ -682,26 +823,50 @@ Map.addLayer(landsat, visParams, ‘Landsat 8 image’);

      2

      Herbaceous

      3

      -

      In the Geometry Tools, click on the marker option (Fig. F2.1.3). This will create a point geometry which will show up as an import named “geometry”. Click on the gear icon to configure this import.

      -

      -

      Fig. F2.1.3 Creating a new layer in the Geometry Imports

      +

      In the Geometry Tools, click on the marker option (Fig. F2.1.3). This will create a point geometry which will show up as an import named “geometry”. Click on the gear icon to configure this import.

      +
      +
      +

      +

      Fig. F2.1.3 Creating a new layer in the Geometry Imports

      +
      +

      We will start by collecting forest points, so name the import forest. Import it as a FeatureCollection, and then click + Property. Name the new property “class” and give it a value of 0 (Fig. F2.1.4). We can also choose a color to represent this class. For a forest class, it is natural to choose a green color. You can choose the color you prefer by clicking on it, or, for more control, you can use a hexadecimal value.

      -

      Hexadecimal values are used throughout the digital world to represent specific colors across computers and operating systems. They are specified by six values arranged in three pairs, with one pair each for the red, green, and blue brightness values. If you’re unfamiliar with hexadecimal values, imagine for a moment that colors were specified in pairs of base 10 numbers instead of pairs of base 16. In that case, a bright pure red value would be “990000”; a bright pure green value would be “009900”; and a bright pure blue value would be “000099”. A value like “501263” would be a mixture of the three colors, not especially bright, having roughly equal amounts of blue and red, and much less green: a color that would be a shade of purple. To create numbers in the hexadecimal system, which might feel entirely natural if humans had evolved to have 16 fingers, sixteen “digits” are needed: a base 16 counter goes 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, A, B, C, D, E, F, then 10, 11, and so on. Given that counting framework, the number “FF” is like “99” in base 10: the largest two-digit number. The hexadecimal color used for coloring the letters of the word FeatureCollection in this book, a color with roughly equal amounts of blue and red, and much less green, is “7F1FA2”  

      -

      Returning to the coloring of the forest points, the hexadecimal value “589400” is a little bit of red, about twice as much green, and no blue: the deep green seen in Figure F2.1.4. Enter that value, with or without the “#” in front, and click OK after finishing the configuration.

      -

      -

      Fig. F2.1.4 Edit geometry layer properties

      -

      Now, in the Geometry Imports, we will see that the import has been renamed forest. Click on it to activate the drawing mode (Fig. F2.1.5) in order to start collecting forest points.

      -

      -

      Fig. F2.1.5 Activate forest layer to start collection

      -

      Now, start collecting points over forested areas (Fig. F2.1.6). Zoom in and out as needed. You can use the satellite basemap to assist you, but the basis of your collection should be the Landsat image. Remember that the more points you collect, the more the classifier will learn from the information you provide. For now, let’s set a goal to collect 25 points per class. Click Exit next to Point drawing (Fig. F2.1.5) when finished.

      -

      -

      Fig. F2.1.6 Forest points

      -

      Repeat the same process for the other classes by creating new layers (Fig. F2.1.7). Don’t forget to import using the FeatureCollection option as mentioned above. For the developed class, collect points over urban areas. For the water class, collect points over the Ligurian Sea, and also look for other bodies of water, like rivers. For the herbaceous class, collect points over agricultural fields. Remember to set the “class” property for each class to its corresponding code (see Table 2.1.1) and click Exit once you finalize collecting points for each class as mentioned above. We will be using the following hexadecimal colors for the other classes: #FF0000 for developed, #1A11FF for water, and #D0741E for herbaceous. 

      -

      -

      Fig. F2.1.7 New layer option in Geometry Imports

      -

      You should now have four FeatureCollection imports named forest, developed, water, and herbaceous (Fig. F2.1.8).

      -

      -

      Fig. F2.1.8 Example of training points

      +

      Hexadecimal values are used throughout the digital world to represent specific colors across computers and operating systems. They are specified by six values arranged in three pairs, with one pair each for the red, green, and blue brightness values. If you’re unfamiliar with hexadecimal values, imagine for a moment that colors were specified in pairs of base 10 numbers instead of pairs of base 16. In that case, a bright pure red value would be “990000”; a bright pure green value would be “009900”; and a bright pure blue value would be “000099”. A value like “501263” would be a mixture of the three colors, not especially bright, having roughly equal amounts of blue and red, and much less green: a color that would be a shade of purple. To create numbers in the hexadecimal system, which might feel entirely natural if humans had evolved to have 16 fingers, sixteen “digits” are needed: a base 16 counter goes 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, A, B, C, D, E, F, then 10, 11, and so on. Given that counting framework, the number “FF” is like “99” in base 10: the largest two-digit number. The hexadecimal color used for coloring the letters of the word FeatureCollection in this book, a color with roughly equal amounts of blue and red, and much less green, is “7F1FA2”

      +

      Returning to the coloring of the forest points, the hexadecimal value “589400” is a little bit of red, about twice as much green, and no blue: the deep green seen in Figure F2.1.4. Enter that value, with or without the “#” in front, and click OK after finishing the configuration.

      +
      +
      +

      +

      Fig. F2.1.4 Edit geometry layer properties

      +
      +
      +

      Now, in the Geometry Imports, we will see that the import has been renamed forest. Click on it to activate the drawing mode (Fig. F2.1.5) in order to start collecting forest points.

      +
      +
      +

      +

      Fig. F2.1.5 Activate forest layer to start collection

      +
      +
      +

      Now, start collecting points over forested areas (Fig. F2.1.6). Zoom in and out as needed. You can use the satellite basemap to assist you, but the basis of your collection should be the Landsat image. Remember that the more points you collect, the more the classifier will learn from the information you provide. For now, let’s set a goal to collect 25 points per class. Click Exit next to Point drawing (Fig. F2.1.5) when finished.

      +
      +
      +

      +

      Fig. F2.1.6 Forest points

      +
      +
      +

      Repeat the same process for the other classes by creating new layers (Fig. F2.1.7). Don’t forget to import using the FeatureCollection option as mentioned above. For the developed class, collect points over urban areas. For the water class, collect points over the Ligurian Sea, and also look for other bodies of water, like rivers. For the herbaceous class, collect points over agricultural fields. Remember to set the “class” property for each class to its corresponding code (see Table 2.1.1) and click Exit once you finalize collecting points for each class as mentioned above. We will be using the following hexadecimal colors for the other classes: #FF0000 for developed, #1A11FF for water, and #D0741E for herbaceous.

      +
      +
      +

      +

      Fig. F2.1.7 New layer option in Geometry Imports

      +
      +
      +

      You should now have four FeatureCollection imports named forest, developed, water, and herbaceous (Fig. F2.1.8).

      +
      +
      +

      +

      Fig. F2.1.8 Example of training points

      +
      +
      @@ -712,84 +877,111 @@ Note
      -

      Code Checkpoint F21a. The book’s repository contains a script that shows what your code should look like at this point.

      +

      Code Checkpoint F21a. The book’s repository contains a script that shows what your code should look like at this point.

      If you wish to have the exact same results demonstrated in this chapter from now on, continue beginning with this Code Checkpoint. If you use the points collected yourself, the results may vary from this point forward.

      -

      The next step is to combine all the training feature collections into one. Copy and paste the code below to combine them into one FeatureCollection called trainingFeatures. Here, we use the flatten method to avoid having a collection of feature collections—we want individual features within our FeatureCollection.

      -

      // Combine training feature collections.
      -var trainingFeatures = ee.FeatureCollection([
      -   forest, developed, water, herbaceous
      -]).flatten();

      -

      Note: Alternatively, you could use an existing set of reference data. For example, the European Space Agency (ESA) WorldCover dataset is a global map of land use and land cover derived from ESA’s Sentinel-2 imagery at 10 m resolution. With existing datasets, we can randomly place points on pixels classified as the classes of interest (if you are curious, you can explore the Earth Engine documentation to learn about the ee.Image.stratifiedSample and the ee.FeatureCollection.randomPoints methods). The drawback is that these global datasets will not always contain the specific classes of interest for your region, or may not be entirely accurate at the local scale. Another option is to use samples that were collected in the field (e.g., GPS points). In Chap. F5.0, you will see how to upload your own data as Earth Engine assets.

      -

      In the combined FeatureCollection, each Feature point should have a property called “class”. The class values are consecutive integers from 0 to 3 (you could verify that this is true by printing trainingFeatures and checking the properties of the features).

      -

      Now that we have our training points, copy and paste the code below to extract the band information for each class at each point location. First, we define the prediction bands to extract different spectral and thermal information from different bands for each class. Then, we use the sampleRegions method to sample the information from the Landsat image at each point location. This method requires information about the FeatureCollection (our reference points), the property to extract (“class”), and the pixel scale (in meters).

      -

      // Define prediction bands.
      -var predictionBands = [   ‘SR_B1’, ‘SR_B2’, ‘SR_B3’, ‘SR_B4’, ‘SR_B5’, ‘SR_B6’, ‘SR_B7’,   ‘ST_B10’
      -];

      -

      // Sample training points.
      -var classifierTraining = landsat.select(predictionBands)
      -   .sampleRegions({
      -       collection: trainingFeatures,
      -       properties: [‘class’],
      -       scale: 30   });

      -

      You can check whether the classifierTraining object extracted the properties of interest by printing it and expanding the first feature. You should see the band and class information (Fig. F2.1.9).

      -

      -

      Fig. F2.1.9 Example of extracted band information for one point of class 0 (forest)

      -

      Now we can choose a classifier. The choice of classifier is not always obvious, and there are many options from which to pick—you can quickly expand the ee.Classifier object under Docs to get an idea of how many options we have for image classification. Therefore, we will be testing different classifiers and comparing their results. We will start with a Classification and Regression Tree (CART) classifier, a well-known classification algorithm (Fig. F2.1.10) that has been around for decades.

      -

      -

      Fig. F2.1.10 Example of a decision tree for satellite image classification. Values and classes are hypothetical.

      -

      Copy and paste the code below to instantiate a CART classifier (ee.Classifier.smileCart) and train it.

      -

      //////////////// CART Classifier ///////////////////

      -

      // Train a CART Classifier.
      -var classifier = ee.Classifier.smileCart().train({
      -   features: classifierTraining,
      -   classProperty: ‘class’,
      -   inputProperties: predictionBands
      -});

      -

      Essentially, the classifier contains the mathematical rules that link labels to spectral information. If you print the variable classifier and expand its properties, you can confirm the basic characteristics of the object (bands, properties, and classifier being used). If you print classifier.explain, you can find a property called “tree” that contains the decision rules.

      +

      The next step is to combine all the training feature collections into one. Copy and paste the code below to combine them into one FeatureCollection called trainingFeatures. Here, we use the flatten method to avoid having a collection of feature collections—we want individual features within our FeatureCollection.

      +
      // Combine training feature collections.  
+var trainingFeatures = ee.FeatureCollection([  
+   forest, developed, water, herbaceous  
+]).flatten();
      +

      Note: Alternatively, you could use an existing set of reference data. For example, the European Space Agency (ESA) WorldCover dataset is a global map of land use and land cover derived from ESA’s Sentinel-2 imagery at 10 m resolution. With existing datasets, we can randomly place points on pixels classified as the classes of interest (if you are curious, you can explore the Earth Engine documentation to learn about the ee.Image.stratifiedSample and the ee.FeatureCollection.randomPoints methods). The drawback is that these global datasets will not always contain the specific classes of interest for your region, or may not be entirely accurate at the local scale. Another option is to use samples that were collected in the field (e.g., GPS points). In Chap. F5.0, you will see how to upload your own data as Earth Engine assets.

      +

      In the combined FeatureCollection, each Feature point should have a property called “class”. The class values are consecutive integers from 0 to 3 (you could verify that this is true by printing trainingFeatures and checking the properties of the features).

      +

      Now that we have our training points, copy and paste the code below to extract the band information for each class at each point location. First, we define the prediction bands to extract different spectral and thermal information from different bands for each class. Then, we use the sampleRegions method to sample the information from the Landsat image at each point location. This method requires information about the FeatureCollection (our reference points), the property to extract (“class”), and the pixel scale (in meters).

      +
      // Define prediction bands.  
+var predictionBands = [   'SR_B1', 'SR_B2', 'SR_B3', 'SR_B4', 'SR_B5', 'SR_B6', 'SR_B7',   'ST_B10'  
+];  
+  
+// Sample training points.  
+var classifierTraining = landsat.select(predictionBands)  
+   .sampleRegions({  
+       collection: trainingFeatures,  
+       properties: ['class'],  
+       scale: 30   });
      +

      You can check whether the classifierTraining object extracted the properties of interest by printing it and expanding the first feature. You should see the band and class information (Fig. F2.1.9).

      +
      +
      +

      +

      Fig. F2.1.9 Example of extracted band information for one point of class 0 (forest)

      +
      +
      +

      Now we can choose a classifier. The choice of classifier is not always obvious, and there are many options from which to pick—you can quickly expand the ee.Classifier object under Docs to get an idea of how many options we have for image classification. Therefore, we will be testing different classifiers and comparing their results. We will start with a Classification and Regression Tree (CART) classifier, a well-known classification algorithm (Fig. F2.1.10) that has been around for decades.

      +
      +
      +

      +

      Fig. F2.1.10 Example of a decision tree for satellite image classification. Values and classes are hypothetical.

      +
      +
      +

      Copy and paste the code below to instantiate a CART classifier (ee.Classifier.smileCart) and train it.

      +
      //////////////// CART Classifier ///////////////////  
+  
+// Train a CART Classifier.  
+var classifier = ee.Classifier.smileCart().train({  
+   features: classifierTraining,  
+   classProperty: 'class',  
+   inputProperties: predictionBands  
+});
      +

      Essentially, the classifier contains the mathematical rules that link labels to spectral information. If you print the variable classifier and expand its properties, you can confirm the basic characteristics of the object (bands, properties, and classifier being used). If you print classifier.explain, you can find a property called “tree” that contains the decision rules.

      After training the classifier, copy and paste the code below to classify the Landsat image and add it to the Map.

      -

      // Classify the Landsat image.
      -var classified = landsat.select(predictionBands).classify(classifier);

      -

      // Define classification image visualization parameters.
      -var classificationVis = {
      -   min: 0,
      -   max: 3,
      -   palette: [‘589400’, ‘ff0000’, ‘1a11ff’, ‘d0741e’]
      -};

      -

      // Add the classified image to the map.
      -Map.addLayer(classified, classificationVis, ‘CART classified’);

      -

      Note that, in the visualization parameters, we define a palette parameter which in this case represents colors for each pixel value (0–3, our class codes). We use the same hexadecimal colors used when creating our training points for each class. This way, we can associate a color with a class when visualizing the classified image in the Map.

      -

      Inspect the result: Activate the Landsat composite layer and the satellite basemap to overlay with the classified images (Fig. F2.1.11). Change the layers’ transparency to inspect some areas. What do you notice? The result might not look very satisfactory in some areas (e.g., confusion between developed and herbaceous classes). Why do you think this is happening? There are a few options to handle misclassification errors:

      +
      // Classify the Landsat image.  
+var classified = landsat.select(predictionBands).classify(classifier);  
+  
+// Define classification image visualization parameters.  
+var classificationVis = {  
+   min: 0,  
+   max: 3,  
+   palette: ['589400', 'ff0000', '1a11ff', 'd0741e']  
+};  
+  
+// Add the classified image to the map.  
+Map.addLayer(classified, classificationVis, 'CART classified');
      +

      Note that, in the visualization parameters, we define a palette parameter which in this case represents colors for each pixel value (0–3, our class codes). We use the same hexadecimal colors used when creating our training points for each class. This way, we can associate a color with a class when visualizing the classified image in the Map.

      +

      Inspect the result: Activate the Landsat composite layer and the satellite basemap to overlay with the classified images (Fig. F2.1.11). Change the layers’ transparency to inspect some areas. What do you notice? The result might not look very satisfactory in some areas (e.g., confusion between developed and herbaceous classes). Why do you think this is happening? There are a few options to handle misclassification errors:

        -
      • Collect more training data We can try incorporating more points to have a more representative sample of the classes.
        • -
      • Tune the model Classifiers typically have “hyperparameters,” which are set to default values. In the case of classification trees, there are ways to tune the number of leaves in the tree, for example. Tuning models is addressed in Chap. F2.2.
        • -
      • Try other classifiers If a classifier’s results are unsatisfying, we can try some of the other classifiers in Earth Engine to see if the result is better or different.
        • -
      • Expand the collection location It is good practice to collect points across the entire image and not just focus on one location. Also, look for pixels of the same class that show variability (e.g., for the developed class, building rooftops look different than house rooftops; for the herbaceous class, crop fields show distinctive seasonality/phenology). 
        • -
      • Add more predictors We can try adding spectral indices to the input variables; this way, we are feeding the classifier new, unique information about each class. For example, there is a good chance that a vegetation index specialized for detecting vegetation health (e.g., NDVI) would improve the developed versus herbaceous classification.
        • +
      • Collect more training data We can try incorporating more points to have a more representative sample of the classes.
        • +
      • Tune the model Classifiers typically have “hyperparameters,” which are set to default values. In the case of classification trees, there are ways to tune the number of leaves in the tree, for example. Tuning models is addressed in Chap. F2.2.
        • +
      • Try other classifiers If a classifier’s results are unsatisfying, we can try some of the other classifiers in Earth Engine to see if the result is better or different.
        • +
      • Expand the collection location It is good practice to collect points across the entire image and not just focus on one location. Also, look for pixels of the same class that show variability (e.g., for the developed class, building rooftops look different than house rooftops; for the herbaceous class, crop fields show distinctive seasonality/phenology).
        • +
      • Add more predictors We can try adding spectral indices to the input variables; this way, we are feeding the classifier new, unique information about each class. For example, there is a good chance that a vegetation index specialized for detecting vegetation health (e.g., NDVI) would improve the developed versus herbaceous classification.
      -

      -

      Fig. F2.1.11 CART classification

      -

      For now, we will try another supervised learning classifier that is widely used: Random Forests (RF). The RF algorithm (Breiman 2001, Pal 2005) builds on the concept of decision trees, but adds strategies to make them more powerful. It is called a “forest” because it operates by constructing a multitude of decision trees. As mentioned previously, a decision tree creates the rules which are used to make decisions. A Random Forest will randomly choose features and make observations, build a forest of decision trees, and then use the full set of trees to estimate the class. It is a great choice when you do not have a lot of insight about the training data.

      -

      -

      Fig. F2.1.12 General concept of Random Forests

      +
      +
      +

      +

      Fig. F2.1.11 CART classification

      +
      +
      +

      For now, we will try another supervised learning classifier that is widely used: Random Forests (RF). The RF algorithm (Breiman 2001, Pal 2005) builds on the concept of decision trees, but adds strategies to make them more powerful. It is called a “forest” because it operates by constructing a multitude of decision trees. As mentioned previously, a decision tree creates the rules which are used to make decisions. A Random Forest will randomly choose features and make observations, build a forest of decision trees, and then use the full set of trees to estimate the class. It is a great choice when you do not have a lot of insight about the training data.

      +
      +
      +

      +

      Fig. F2.1.12 General concept of Random Forests

      +
      +

      Copy and paste the code below to train the RF classifier (ee.Classifier.smileRandomForest) and apply the classifier to the image. The RF algorithm requires, as its argument, the number of trees to build. We will use 50 trees.

      -

      /////////////// Random Forest Classifier /////////////////////

      -

      // Train RF classifier.
      -var RFclassifier = ee.Classifier.smileRandomForest(50).train({
      -   features: classifierTraining,
      -   classProperty: ‘class’,
      -   inputProperties: predictionBands
      -});

      -

      // Classify Landsat image.
      -var RFclassified = landsat.select(predictionBands).classify(
      -   RFclassifier);

      -

      // Add classified image to the map.
      -Map.addLayer(RFclassified, classificationVis, ‘RF classified’);

      -

      Note that in the ee.Classifier.smileRandomForest documentation (Docs tab), there is a seed (random number) parameter. Setting a seed allows you to exactly replicate your model each time you run it. Any number is acceptable as a seed.

      +
      /////////////// Random Forest Classifier /////////////////////  
+  
+// Train RF classifier.  
+var RFclassifier = ee.Classifier.smileRandomForest(50).train({  
+   features: classifierTraining,  
+   classProperty: 'class',  
+   inputProperties: predictionBands  
+});  
+  
+// Classify Landsat image.  
+var RFclassified = landsat.select(predictionBands).classify(  
+   RFclassifier);  
+  
+// Add classified image to the map.  
+Map.addLayer(RFclassified, classificationVis, 'RF classified');
      +

      Note that in the ee.Classifier.smileRandomForest documentation (Docs tab), there is a seed (random number) parameter. Setting a seed allows you to exactly replicate your model each time you run it. Any number is acceptable as a seed.

      Inspect the result (Fig. F2.1.13). How does this classified image differ from the CART one? Is the classifications better or worse? Zoom in and out and change the transparency of layers as needed. In Chap. F2.2, you will see more systematic ways to assess what is better or worse, based on accuracy metrics.

      -

      -

      Fig. F2.1.13 Random Forest classified image

      +
      +
      +

      +

      Fig. F2.1.13 Random Forest classified image

      +
      +
      @@ -800,14 +992,14 @@ Note
      -

      Code Checkpoint F21b. The book’s repository contains a script that shows what your code should look like at this point.

      +

      Code Checkpoint F21b. The book’s repository contains a script that shows what your code should look like at this point.

      6.2 Unsupervised Classification

      -

      In an unsupervised classification, we have the opposite process of supervised classification. Spectral classes are grouped first and then categorized into clusters. Therefore, in Earth Engine, these classifiers are ee.Clusterer objects. They are “self-taught” algorithms that do not use a set of labeled training data (i.e., they are “unsupervised”). You can think of it as performing a task that you have not experienced before, starting by gathering as much information as possible. For example, imagine learning a new language without knowing the basic grammar, learning only by watching a TV series in that language, listening to examples, and finding patterns.

      -

      Similar to the supervised classification, unsupervised classification in Earth Engine has this workflow:

      +

      In an unsupervised classification, we have the opposite process of supervised classification. Spectral classes are grouped first and then categorized into clusters. Therefore, in Earth Engine, these classifiers are ee.Clusterer objects. They are “self-taught” algorithms that do not use a set of labeled training data (i.e., they are “unsupervised”). You can think of it as performing a task that you have not experienced before, starting by gathering as much information as possible. For example, imagine learning a new language without knowing the basic grammar, learning only by watching a TV series in that language, listening to examples, and finding patterns.

      +

      Similar to the supervised classification, unsupervised classification in Earth Engine has this workflow:

      1. Assemble features with numeric properties in which to find clusters (training data).
      2. Select and instantiate a clusterer.
        3. @@ -815,29 +1007,39 @@ Note
      3. Apply the clusterer to the scene (classification).
      4. Label the clusters.
      -

      In order to generate training data, we will use the sample method, which randomly takes samples from a region (unlike sampleRegions, which takes samples from predefined locations). We will use the image’s footprint as the region by calling the geometry method. Additionally, we will define the number of pixels (numPixels) to sample—in this case, 1000 pixels—and define a tileScale of 8 to avoid computation errors due to the size of the region. Copy and paste the code below to sample 1000 pixels from the Landsat image. You should add to the same script as before to compare supervised versus unsupervised classification results at the end.

      -

      //////////////// Unsupervised classification ////////////////

      -

      // Make the training dataset.
      -var training = landsat.sample({
      -   region: landsat.geometry(),
      -   scale: 30,
      -   numPixels: 1000,
      -   tileScale: 8
      -});

      +

      In order to generate training data, we will use the sample method, which randomly takes samples from a region (unlike sampleRegions, which takes samples from predefined locations). We will use the image’s footprint as the region by calling the geometry method. Additionally, we will define the number of pixels (numPixels) to sample—in this case, 1000 pixels—and define a tileScale of 8 to avoid computation errors due to the size of the region. Copy and paste the code below to sample 1000 pixels from the Landsat image. You should add to the same script as before to compare supervised versus unsupervised classification results at the end.

      +
      //////////////// Unsupervised classification ////////////////  
+  
+// Make the training dataset.  
+var training = landsat.sample({  
+   region: landsat.geometry(),  
+   scale: 30,  
+   numPixels: 1000,  
+   tileScale: 8  
+});

      Now we can instantiate a clusterer and train it. As with the supervised algorithms, there are many unsupervised algorithms to choose from. We will use the k-means clustering algorithm, which is a commonly used approach in remote sensing. This algorithm identifies groups of pixels near each other in the spectral space (image x bands) by using an iterative regrouping strategy. We define a number of clusters, k, and then the method randomly distributes that number of seed points into the spectral space. A large sample of pixels is then grouped into its closest seed, and the mean spectral value of this group is calculated. That mean value is akin to a center of mass of the points, and is known as the centroid. Each iteration recalculates the class means and reclassifies pixels with respect to the new means. This process is repeated until the centroids remain relatively stable and only a few pixels change from class to class on subsequent iterations.

      -

      -

      Fig. F2.1.14 K-means visual concept

      -

      Copy and paste the code below to request four clusters, the same number as for the supervised classification, in order to directly compare them.

      -

      // Instantiate the clusterer and train it.
      -var clusterer = ee.Clusterer.wekaKMeans(4).train(training);

      -

      Now copy and paste the code below to apply the clusterer to the image and add the resulting classification to the Map (Fig. F2.1.15). Note that we are using a method called randomVisualizer to assign colors for the visualization. We are not associating the unsupervised classes with the color palette we defined earlier in the supervised classification. Instead, we are assigning random colors to the classes, since we do not yet know which of the unsupervised classes best corresponds to each of the named classes (e.g., forest , herbaceous). Note that the colors in Fig. F1.2.15 might not be the same as you see on your Map, since they are assigned randomly.

      -

      // Cluster the input using the trained clusterer.
      -var Kclassified = landsat.cluster(clusterer);

      -

      // Display the clusters with random colors.
      -Map.addLayer(Kclassified.randomVisualizer(), {},   ‘K-means classified - random colors’);

      -

      -

      Fig. F2.1.15 K-means classification

      -

      Inspect the results. How does this classification compare to the previous ones? If preferred, use the Inspector to check which classes were assigned to each pixel value (“cluster” band) and change the last line of your code to apply the same palette used for the supervised classification results (see Code Checkpoint below for an example).

      +
      +
      +

      +

      Fig. F2.1.14 K-means visual concept

      +
      +
      +

      Copy and paste the code below to request four clusters, the same number as for the supervised classification, in order to directly compare them.

      +
      // Instantiate the clusterer and train it.  
+var clusterer = ee.Clusterer.wekaKMeans(4).train(training);
      +

      Now copy and paste the code below to apply the clusterer to the image and add the resulting classification to the Map (Fig. F2.1.15). Note that we are using a method called randomVisualizer to assign colors for the visualization. We are not associating the unsupervised classes with the color palette we defined earlier in the supervised classification. Instead, we are assigning random colors to the classes, since we do not yet know which of the unsupervised classes best corresponds to each of the named classes (e.g., forest , herbaceous). Note that the colors in Fig. F1.2.15 might not be the same as you see on your Map, since they are assigned randomly.

      +
      // Cluster the input using the trained clusterer.  
+var Kclassified = landsat.cluster(clusterer);  
+  
+// Display the clusters with random colors.  
+Map.addLayer(Kclassified.randomVisualizer(), {},   'K-means classified - random colors');
      +
      +
      +

      +

      Fig. F2.1.15 K-means classification

      +
      +
      +

      Inspect the results. How does this classification compare to the previous ones? If preferred, use the Inspector to check which classes were assigned to each pixel value (“cluster” band) and change the last line of your code to apply the same palette used for the supervised classification results (see Code Checkpoint below for an example).

      Another key point of classification is the accuracy assessment of the results. This will be covered in Chap. F2.2.

      @@ -849,7 +1051,7 @@ Note
      -

      Code Checkpoint F21c. The book’s repository contains a script that shows what your code should look like at this point.

      +

      Code Checkpoint F21c. The book’s repository contains a script that shows what your code should look like at this point.

      @@ -857,11 +1059,11 @@ Note

      Synthesis

      Test if you can improve the classifications by completing the following assignments.

      Assignment 1. For the supervised classification, try collecting more points for each class. The more points you have, the more spectrally represented the classes are. It is good practice to collect points across the entire composite and not just focus on one location. Also look for pixels of the same class that show variability. For example, for the water class, collect pixels in parts of rivers that vary in color. For the developed class, collect pixels from different rooftops.

      -

      Assignment 2. Add more predictors. Usually, the more spectral information you feed the classifier, the easier it is to separate classes. Try calculating and incorporating a band of  NDVI or the Normalized Difference Water Index (Chap. F2.0) as a predictor band. Does this help the classification? Check for developed areas that were being classified as herbaceous or vice versa.

      -

      Assignment 3. Use more trees in the Random Forest classifier. Do you see any improvements compared to 50 trees? Note that the more trees you have, the longer it will take to compute the results, and that more trees might not always mean better results.

      +

      Assignment 2. Add more predictors. Usually, the more spectral information you feed the classifier, the easier it is to separate classes. Try calculating and incorporating a band of NDVI or the Normalized Difference Water Index (Chap. F2.0) as a predictor band. Does this help the classification? Check for developed areas that were being classified as herbaceous or vice versa.

      +

      Assignment 3. Use more trees in the Random Forest classifier. Do you see any improvements compared to 50 trees? Note that the more trees you have, the longer it will take to compute the results, and that more trees might not always mean better results.

      Assignment 4. Increase the number of samples that are extracted from the composite in the unsupervised classification. Does that improve the result?

      -

      Assignment 5. Increase the number k of clusters for the k-means algorithm. What would happen if you tried 10 classes? Does the classified map result in meaningful classes?

      -

      Assignment 6. Test other clustering algorithms. We only used k-means; try other options under the ee.Clusterer object.

      +

      Assignment 5. Increase the number k of clusters for the k-means algorithm. What would happen if you tried 10 classes? Does the classified map result in meaningful classes?

      +

      Assignment 6. Test other clustering algorithms. We only used k-means; try other options under the ee.Clusterer object.

      Conclusion

      @@ -912,7 +1114,7 @@ Chapter Information

      Assumes you know how to:

        -
      • ​​Create a graph using ui.Chart (Chap. F1.3).
        • +
      • ​​Create a graph using ui.Chart (Chap. F1.3).
      • Perform a supervised Random Forest image classification (Chap. F2.1).
      @@ -922,14 +1124,14 @@ Chapter Information

      Introduction

      Any map or remotely sensed product is a generalization or model that will have inherent errors. Products derived from remotely sensed data used for scientific purposes and policymaking require a quantitative measure of accuracy to strengthen the confidence in the information generated (Foody 2002, Strahler et al. 2006, Olofsson et al. 2014). Accuracy assessment is a crucial part of any classification project, as it measures the degree to which the classification agrees with another data source that is considered to be accurate, ground-truth data (i.e., “reality”).

      The history of accuracy assessment reveals increasing detail and rigor in the analysis, moving from a basic visual appraisal of the derived map (Congalton 1994, Foody 2002) to the definition of best practices for sampling and response designs and the calculation of accuracy metrics (Foody 2002, Stehman 2013, Olofsson et al. 2014, Stehman and Foody 2019). The confusion matrix (also called the “error matrix”) (Stehman 1997) summarizes key accuracy metrics used to assess products derived from remotely sensed data.

      -

      In Chap. F2.1, we asked whether the classification results were satisfactory. In remote sensing, the quantification of the answer to that question is called accuracy assessment. In the classification context, accuracy measurements are often derived from a confusion matrix.

      -

      In a thorough accuracy assessment, we think carefully about the sampling design, the response design, and the analysis (Olofsson et al. 2014). Fundamental protocols are taken into account to produce scientifically rigorous and transparent estimates of accuracy and area, which requires robust planning and time. In a standard setting, we would calculate the number of samples needed for measuring accuracy (sampling design). Here, we will focus mainly on the last step, analysis, by examining the confusion matrix and learning how to calculate the accuracy metrics. This will be done by partitioning the existing data into training and testing sets.

      +

      In Chap. F2.1, we asked whether the classification results were satisfactory. In remote sensing, the quantification of the answer to that question is called accuracy assessment. In the classification context, accuracy measurements are often derived from a confusion matrix.

      +

      In a thorough accuracy assessment, we think carefully about the sampling design, the response design, and the analysis (Olofsson et al. 2014). Fundamental protocols are taken into account to produce scientifically rigorous and transparent estimates of accuracy and area, which requires robust planning and time. In a standard setting, we would calculate the number of samples needed for measuring accuracy (sampling design). Here, we will focus mainly on the last step, analysis, by examining the confusion matrix and learning how to calculate the accuracy metrics. This will be done by partitioning the existing data into training and testing sets.

      7.1 Quantifying Classification Accuracy Through a Confusion Matrix

      -

      If you have not already done so, be sure to add the book’s code repository to the Code Editor by entering https://code.earthengine.google.com/?accept_repo=projects/gee-edu/book into your browser. The book’s scripts will then be available in the script manager panel. If you have trouble finding the repo, you can visit this link for help.

      -

      To illustrate some of the basic ideas about classification accuracy, we will revisit the data and location of part of Chap. F2.1, where we tested different classifiers and classified a Landsat image of the area around Milan, Italy. We will name this dataset ‘data’. This variable is a FeatureCollection with features containing the “class” values (Table F2.2.1) and spectral information of four land cover / land use classes: forest, developed, water, and herbaceous (see Fig. F2.1.8 and Fig. F2.1.9 for a refresher). We will also define a variable, predictionBands, which is a list of bands that will be used for prediction (classification)—the spectral information in the data variable.

      -

      Table F2.2.1 Land cover classes

      +

      If you have not already done so, be sure to add the book’s code repository to the Code Editor by entering https://code.earthengine.google.com/?accept_repo=projects/gee-edu/book into your browser. The book’s scripts will then be available in the script manager panel. If you have trouble finding the repo, you can visit this link for help.

      +

      To illustrate some of the basic ideas about classification accuracy, we will revisit the data and location of part of Chap. F2.1, where we tested different classifiers and classified a Landsat image of the area around Milan, Italy. We will name this dataset ‘data’. This variable is a FeatureCollection with features containing the “class” values (Table F2.2.1) and spectral information of four land cover / land use classes: forest, developed, water, and herbaceous (see Fig. F2.1.8 and Fig. F2.1.9 for a refresher). We will also define a variable, predictionBands, which is a list of bands that will be used for prediction (classification)—the spectral information in the data variable.

      +

      Table F2.2.1 Land cover classes

      Class

      Class value

      Forest

      @@ -940,38 +1142,40 @@ Chapter Information

      2

      Herbaceous

      3

      -

      The first step is to partition the set of known values into training and testing sets in order to have something for the classifier to predict over that it has not been shown before (the testing set), mimicking unseen data that the model might see in the future. We add a column of random numbers to our FeatureCollection using the randomColumn method. Then, we filter the features into about 80% for training and 20% for testing using ee.Filter. Copy and paste the code below to partition the data and filter features based on the random number.

      -

      // Import the reference dataset.
      -var data = ee.FeatureCollection(   ‘projects/gee-book/assets/F2-2/milan_data’);

      -

      // Define the prediction bands.
      -var predictionBands = [   ‘SR_B1’, ‘SR_B2’, ‘SR_B3’, ‘SR_B4’, ‘SR_B5’, ‘SR_B6’, ‘SR_B7’,   ‘ST_B10’,   ‘ndvi’, ‘ndwi’
      -];

      -

      // Split the dataset into training and testing sets.
      -var trainingTesting = data.randomColumn();
      -var trainingSet = trainingTesting
      -   .filter(ee.Filter.lessThan(‘random’, 0.8));
      -var testingSet = trainingTesting
      -   .filter(ee.Filter.greaterThanOrEquals(‘random’, 0.8));

      -

      Note that randomColumn creates pseudorandom numbers in a deterministic way. This makes it possible to generate a reproducible pseudorandom sequence by defining the seed parameter (Earth Engine uses a seed of 0 by default). In other words, given a starting value (i.e., the seed), randomColumn will always provide the same sequence of pseudorandom numbers.

      +

      The first step is to partition the set of known values into training and testing sets in order to have something for the classifier to predict over that it has not been shown before (the testing set), mimicking unseen data that the model might see in the future. We add a column of random numbers to our FeatureCollection using the randomColumn method. Then, we filter the features into about 80% for training and 20% for testing using ee.Filter. Copy and paste the code below to partition the data and filter features based on the random number.

      +
      // Import the reference dataset.  
+var data = ee.FeatureCollection(   'projects/gee-book/assets/F2-2/milan_data');  
+  
+// Define the prediction bands.  
+var predictionBands = [   'SR_B1', 'SR_B2', 'SR_B3', 'SR_B4', 'SR_B5', 'SR_B6', 'SR_B7',   'ST_B10',   'ndvi', 'ndwi'  
+];  
+  
+// Split the dataset into training and testing sets.  
+var trainingTesting = data.randomColumn();  
+var trainingSet = trainingTesting  
+   .filter(ee.Filter.lessThan('random', 0.8));  
+var testingSet = trainingTesting  
+   .filter(ee.Filter.greaterThanOrEquals('random', 0.8));
      +

      Note that randomColumn creates pseudorandom numbers in a deterministic way. This makes it possible to generate a reproducible pseudorandom sequence by defining the seed parameter (Earth Engine uses a seed of 0 by default). In other words, given a starting value (i.e., the seed), randomColumn will always provide the same sequence of pseudorandom numbers.

      Copy and paste the code below to train a Random Forest classifier with 50 decision trees using the trainingSet.

      -

      // Train the Random Forest Classifier with the trainingSet.
      -var RFclassifier = ee.Classifier.smileRandomForest(50).train({
      -   features: trainingSet,
      -   classProperty: ‘class’,
      -   inputProperties: predictionBands
      -});

      +
      // Train the Random Forest Classifier with the trainingSet.  
+var RFclassifier = ee.Classifier.smileRandomForest(50).train({  
+   features: trainingSet,  
+   classProperty: 'class',  
+   inputProperties: predictionBands  
+});

      Now, let’s discuss what a confusion matrix is. A confusion matrix describes the quality of a classification by comparing the predicted values to the actual values. A simple example is a confusion matrix for a binary classification into the classes “positive” and “negative,” as shown in Table F2.2.1.

      -

      Table F2.2.1 Confusion matrix for a binary classification where the classes are “positive” and “negative”

      +

      Table F2.2.1 Confusion matrix for a binary classification where the classes are “positive” and “negative”

      Actual values

      Positive

      Negative

      Predicted values

      Positive

      -

      TP (true positive)

      -

      FP (false positive)

      +

      TP (true positive)

      +

      FP (false positive)

      Negative

      -

      FN (false negative)

      -

      TN (true negative)

      +

      FN (false negative)

      +

      TN (true negative)

      In Table F2.2.1, the columns represent the actual values (the truth), while the rows represent the predictions (the classification). “True positive” (TP) and “true negative” (TN) mean that the classification of a pixel matches the truth (e.g., a water pixel correctly classified as water). “False positive” (FP) and “false negative” (FN) mean that the classification of a pixel does not match the truth (e.g., a non-water pixel incorrectly classified as water).

      • TP: classified as positive and the actual class is positive
        • @@ -980,7 +1184,7 @@ var RFclassifier = ee.Classifier.smileRandomForest(50).train({
      • TN: classified as negative and the actual class is negative

      We can extract some statistical information from a confusion matrix.. Let’s look at an example to make this clearer. Table F2.2.2 is a confusion matrix for a sample of 1,000 pixels for a classifier that identifies whether a pixel is forest (positive) or non-forest (negative), a binary classification.

      -

      Table F2.2.2 Confusion matrix for a binary classification where the classes are “positive” (forest) and “negative” (non-forest)

      +

      Table F2.2.2 Confusion matrix for a binary classification where the classes are “positive” (forest) and “negative” (non-forest)

      Actual values

      Positive

      Negative

      @@ -1003,34 +1207,38 @@ var RFclassifier = ee.Classifier.smileRandomForest(50).train({

      The user’s accuracy (also called the “consumer’s accuracy”) is the accuracy of the map from the point of view of a map user, and is calculated as the number of correctly identified pixels of a given class divided by the total number of pixels claimed to be in that class. The user’s accuracy for a given class tells us the proportion of the pixels identified on the map as being in that class that are actually in that class on the ground.

      -

      In this case, the user’s accuracy for the forest class is 94.5%, calculated using ). The user’s accuracy for the non-forest class is 97.9%, calculated from ).

      +

      In this case, the user’s accuracy for the forest class is 94.5%, calculated using ). The user’s accuracy for the non-forest class is 97.9%, calculated from ).

      Fig. F2.2.1 helps visualize the rows and columns used to calculate each accuracy.

      -

      -

      Fig. F2.2.1 Confusion matrix for a binary classification where the classes are “positive” (forest) and “negative” (non-forest), with accuracy metrics

      +
      +
      +

      +

      Fig. F2.2.1 Confusion matrix for a binary classification where the classes are “positive” (forest) and “negative” (non-forest), with accuracy metrics

      +
      +

      It is very common to talk about two types of error when addressing remote-sensing classification accuracy: omission errors and commission errors. Omission errors refer to the reference pixels that were left out of (omitted from) the correct class in the classified map. In a two-class system, an error of omission in one class will be counted as an error of commission in another class. Omission errors are complementary to the producer’s accuracy.

      -

       

      +

      Commission errors refer to the class pixels that were erroneously classified in the map and are complementary to the user’s accuracy.

      -

      Finally, another commonly used accuracy metric is the kappa coefficient, which  evaluates how well the classification performed as compared to random. The value of the kappa coefficient can range from −1 to 1: a negative value indicates that the classification is worse than a random assignment of categories would have been; a value of 0 indicates that the classification is no better or worse than random; and a positive value indicates that the classification is better than random.

      +

      Finally, another commonly used accuracy metric is the kappa coefficient, which evaluates how well the classification performed as compared to random. The value of the kappa coefficient can range from −1 to 1: a negative value indicates that the classification is worse than a random assignment of categories would have been; a value of 0 indicates that the classification is no better or worse than random; and a positive value indicates that the classification is better than random.

      The chance agreement is calculated as the sum of the product of row and column totals for each class, and the observed accuracy is the overall accuracy. Therefore, for our example, the kappa coefficient is 0.927.

      Now, let’s go back to the script. In Earth Engine, there are API calls for these operations. Note that our confusion matrix will be a 4 x 4 table, since we have four different classes.

      -

      Copy and paste the code below to classify the testingSet and get a confusion matrix using the method errorMatrix. Note that the classifier automatically adds a property called “classification,” which is compared to the “class” property of the reference dataset.

      -

      // Now, to test the classification (verify model’s accuracy),
      -// we classify the testingSet and get a confusion matrix.
      -var confusionMatrix = testingSet.classify(RFclassifier)
      -   .errorMatrix({
      -       actual: ‘class’,
      -       predicted: ‘classification’   });

      -

      Copy and paste the code below to print the confusion matrix and accuracy metrics. Expand the confusion matrix object to inspect it. The entries represent the number of pixels. Items on the diagonal represent correct classification. Items off the diagonal are misclassifications, where the class in row i is classified as column j (values from 0 to 3 correspond to our class codes: forest, developed, water, and herbaceous, respectively). Also expand the producer’s accuracy, user’s accuracy (consumer’s accuracy), and kappa coefficient objects to inspect them.

      -

      // Print the results.
      -print(‘Confusion matrix:’, confusionMatrix);
      -print(‘Overall Accuracy:’, confusionMatrix.accuracy());
      -print(‘Producers Accuracy:’, confusionMatrix.producersAccuracy());
      -print(‘Consumers Accuracy:’, confusionMatrix.consumersAccuracy());
      -print(‘Kappa:’, confusionMatrix.kappa());

      -

      How is the classification accuracy? Which classes have higher accuracy compared to the others? Can you think of any reasons why? (Hint: Check where the errors in these classes are in the confusion matrix—i.e., being committed and omitted.)

      +

      Copy and paste the code below to classify the testingSet and get a confusion matrix using the method errorMatrix. Note that the classifier automatically adds a property called “classification,” which is compared to the “class” property of the reference dataset.

      +
      // Now, to test the classification (verify model's accuracy),  
+// we classify the testingSet and get a confusion matrix.  
+var confusionMatrix = testingSet.classify(RFclassifier)  
+   .errorMatrix({  
+       actual: 'class',  
+       predicted: 'classification'   });
      +

      Copy and paste the code below to print the confusion matrix and accuracy metrics. Expand the confusion matrix object to inspect it. The entries represent the number of pixels. Items on the diagonal represent correct classification. Items off the diagonal are misclassifications, where the class in row i is classified as column j (values from 0 to 3 correspond to our class codes: forest, developed, water, and herbaceous, respectively). Also expand the producer’s accuracy, user’s accuracy (consumer’s accuracy), and kappa coefficient objects to inspect them.

      +
      // Print the results.  
+print('Confusion matrix:', confusionMatrix);  
+print('Overall Accuracy:', confusionMatrix.accuracy());  
+print('Producers Accuracy:', confusionMatrix.producersAccuracy());  
+print('Consumers Accuracy:', confusionMatrix.consumersAccuracy());  
+print('Kappa:', confusionMatrix.kappa());
      +

      How is the classification accuracy? Which classes have higher accuracy compared to the others? Can you think of any reasons why? (Hint: Check where the errors in these classes are in the confusion matrix—i.e., being committed and omitted.)

      @@ -1041,38 +1249,44 @@ Note
      -

      Code Checkpoint F22a. The book’s repository contains a script that shows what your code should look like at this point.

      +

      Code Checkpoint F22a. The book’s repository contains a script that shows what your code should look like at this point.

      7.2 Hyperparameter tuning

      We can also assess how the number of trees in the Random Forest classifier affects the classification accuracy. Copy and paste the code below to create a function that charts the overall accuracy versus the number of trees used. The code tests from 5 to 100 trees at increments of 5, producing Fig. F2.2.2. (Do not worry too much about fully understanding each item at this stage of your learning. If you want to find out how these operations work, you can see more in Chaps. F4.0 and F4.1.)

      -

      // Hyperparameter tuning.
      -var numTrees = ee.List.sequence(5, 100, 5);

      -

      var accuracies = numTrees.map(function(t) {   var classifier = ee.Classifier.smileRandomForest(t)
      -       .train({
      -           features: trainingSet,
      -           classProperty: ‘class’,
      -           inputProperties: predictionBands
      -       });   return testingSet
      -       .classify(classifier)
      -       .errorMatrix(‘class’, ‘classification’)
      -       .accuracy();
      -});

      -

      print(ui.Chart.array.values({
      -   array: ee.Array(accuracies),
      -   axis: 0,
      -   xLabels: numTrees
      -}).setOptions({
      -   hAxis: {
      -       title: ‘Number of trees’   },
      -   vAxis: {
      -       title: ‘Accuracy’   },
      -   title: ‘Accuracy per number of trees’
      -}));

      -

      -

      Fig. F2.2.2 Chart showing accuracy per number of Random Forest trees

      +
      // Hyperparameter tuning.  
+var numTrees = ee.List.sequence(5, 100, 5);  
+  
+var accuracies = numTrees.map(function(t) {   var classifier = ee.Classifier.smileRandomForest(t)  
+       .train({  
+           features: trainingSet,  
+           classProperty: 'class',  
+           inputProperties: predictionBands  
+       });   return testingSet  
+       .classify(classifier)  
+       .errorMatrix('class', 'classification')  
+       .accuracy();  
+});  
+  
+print(ui.Chart.array.values({  
+   array: ee.Array(accuracies),  
+   axis: 0,  
+   xLabels: numTrees  
+}).setOptions({  
+   hAxis: {  
+       title: 'Number of trees'   },  
+   vAxis: {  
+       title: 'Accuracy'   },  
+   title: 'Accuracy per number of trees'  
+}));
      +
      +
      +

      +

      Fig. F2.2.2 Chart showing accuracy per number of Random Forest trees

      +
      +
      @@ -1083,10 +1297,10 @@ Note
      -

      Code Checkpoint F22b. The book’s repository contains a script that shows what your code should look like at this point.

      +

      Code Checkpoint F22b. The book’s repository contains a script that shows what your code should look like at this point.

      -

      Section 3. Spatial autocorrelation

      +

      Section 3. Spatial autocorrelation

      We might also want to ensure that the samples from the training set are uncorrelated with the samples from the testing set. This might result from the spatial autocorrelation of the phenomenon being predicted. One way to exclude samples that might be correlated in this manner is to remove samples that are within some distance to any other sample. In Earth Engine, this can be accomplished with a spatial join. The following Code Checkpoint replicates Sect. 1 but with a spatial join that excludes training points that are less than 1000 meters distant from testing points.

      @@ -1098,19 +1312,19 @@ Note
      -

      Code Checkpoint F22c. The book’s repository contains a script that shows what your code should look like at this point.

      +

      Code Checkpoint F22c. The book’s repository contains a script that shows what your code should look like at this point.

      Synthesis

      -

      Assignment 1. Based on Sect. 1, test other classifiers (e.g., a Classification and Regression Tree or Support Vector Machine classifier) and compare the accuracy results with the Random Forest results. Which model performs better?

      -

      Assignment 2. Try setting a different seed in the randomColumn method and see how that affects the accuracy results. You can also change the split between the training and testing sets (e.g., 70/30 or 60/40).

      +

      Assignment 1. Based on Sect. 1, test other classifiers (e.g., a Classification and Regression Tree or Support Vector Machine classifier) and compare the accuracy results with the Random Forest results. Which model performs better?

      +

      Assignment 2. Try setting a different seed in the randomColumn method and see how that affects the accuracy results. You can also change the split between the training and testing sets (e.g., 70/30 or 60/40).

      Conclusion

      You should now understand how to calculate how well your classifier is performing on the data used to build the model. This is a useful way to understand how a classifier is performing, because it can help indicate which classes are performing better than others. A poorly modeled class can sometimes be improved by, for example, collecting more training points for that class.

      -

      Nevertheless, a model may work well on training data but work poorly in locations randomly chosen in the study area. To understand a model’s behavior on testing data, analysts employ protocols required to produce scientifically rigorous and transparent estimates of the accuracy and area of each class in the study region. We will not explore those practices in this chapter, but if you are interested, there are tutorials and papers available online that can guide you through the process. Links to some of those tutorials can be found in the “For Further Reading” section of this book.

      +

      Nevertheless, a model may work well on training data but work poorly in locations randomly chosen in the study area. To understand a model’s behavior on testing data, analysts employ protocols required to produce scientifically rigorous and transparent estimates of the accuracy and area of each class in the study region. We will not explore those practices in this chapter, but if you are interested, there are tutorials and papers available online that can guide you through the process. Links to some of those tutorials can be found in the “For Further Reading” section of this book.

      References

      Congalton R (1994) Accuracy assessment of remotely sensed data: Future needs and directions. In: Proceedings of Pecora 12 land information from space-based systems. pp 385–388

      Foody GM (2002) Status of land cover classification accuracy assessment. Remote Sens Environ 80:185–201. https://doi.org/10.1016/S0034-4257(01)00295-4

      diff --git a/docs/F4.html b/docs/F4.html index 4da1a86..ba8551b 100644 --- a/docs/F4.html +++ b/docs/F4.html @@ -20,6 +20,69 @@ ul.task-list li input[type="checkbox"] { margin: 0 0.8em 0.2em -1.6em; vertical-align: middle; } +pre > code.sourceCode { white-space: pre; position: relative; } +pre > code.sourceCode > span { display: inline-block; line-height: 1.25; } +pre > code.sourceCode > span:empty { height: 1.2em; } +.sourceCode { overflow: visible; } +code.sourceCode > span { color: inherit; text-decoration: inherit; } +div.sourceCode { margin: 1em 0; } +pre.sourceCode { margin: 0; } +@media screen { +div.sourceCode { overflow: auto; } +} +@media print { +pre > code.sourceCode { white-space: pre-wrap; } +pre > code.sourceCode > span { text-indent: -5em; padding-left: 5em; } +} +pre.numberSource code + { counter-reset: source-line 0; } +pre.numberSource code > span + { position: relative; left: -4em; counter-increment: source-line; } +pre.numberSource code > span > a:first-child::before + { content: counter(source-line); + position: relative; left: -1em; text-align: right; vertical-align: baseline; + border: none; display: inline-block; + -webkit-touch-callout: none; -webkit-user-select: none; + -khtml-user-select: none; -moz-user-select: none; + -ms-user-select: none; user-select: none; + padding: 0 4px; width: 4em; + color: #aaaaaa; + } +pre.numberSource { margin-left: 3em; border-left: 1px solid #aaaaaa; padding-left: 4px; } +div.sourceCode + { } +@media screen { +pre > code.sourceCode > span > a:first-child::before { text-decoration: underline; } +} +code span.al { color: #ff0000; font-weight: bold; } /* Alert */ +code span.an { color: #60a0b0; font-weight: bold; font-style: italic; } /* Annotation */ +code span.at { color: #7d9029; } /* Attribute */ +code span.bn { color: #40a070; } /* BaseN */ +code span.bu { color: #008000; } /* BuiltIn */ +code span.cf { color: #007020; font-weight: bold; } /* ControlFlow */ +code span.ch { color: #4070a0; } /* Char */ +code span.cn { color: #880000; } /* Constant */ +code span.co { color: #60a0b0; font-style: italic; } /* Comment */ +code span.cv { color: #60a0b0; font-weight: bold; font-style: italic; } /* CommentVar */ +code span.do { color: #ba2121; font-style: italic; } /* Documentation */ +code span.dt { color: #902000; } /* DataType */ +code span.dv { color: #40a070; } /* DecVal */ +code span.er { color: #ff0000; font-weight: bold; } /* Error */ +code span.ex { } /* Extension */ +code span.fl { color: #40a070; } /* Float */ +code span.fu { color: #06287e; } /* Function */ +code span.im { color: #008000; font-weight: bold; } /* Import */ +code span.in { color: #60a0b0; font-weight: bold; font-style: italic; } /* Information */ +code span.kw { color: #007020; font-weight: bold; } /* Keyword */ +code span.op { color: #666666; } /* Operator */ +code span.ot { color: #007020; } /* Other */ +code span.pp { color: #bc7a00; } /* Preprocessor */ +code span.sc { color: #4070a0; } /* SpecialChar */ +code span.ss { color: #bb6688; } /* SpecialString */ +code span.st { color: #4070a0; } /* String */ +code span.va { color: #19177c; } /* Variable */ +code span.vs { color: #4070a0; } /* VerbatimString */ +code span.wa { color: #60a0b0; font-weight: bold; font-style: italic; } /* Warning */ @@ -238,12 +301,14 @@ gtag('config', 'G-RK9ZLZQ6GL', { 'anonymize_ip': true});
    • 7 Exploring Image Collections
      • -
    • 8 Aggregating Images for Time Series +
    • 8 Aggregating Images for Time Series
    • 9 Clouds and Image Compositing
        -
      • 9.1 Incorporating Data from Other Satellites
        • -
      • 9.2 Best-Available-Pixel Compositing Earth Engine Application
        • +
      • 9.1 Cloud Filter and Cloud Mask
        • +
      • 9.2 Incorporating Data from Other Satellites
        • +
      • 9.3 Best-Available-Pixel Compositing Earth Engine Application
      • Synthesis
      • Conclusion
      • References
        • @@ -284,30 +350,19 @@ gtag('config', 'G-RK9ZLZQ6GL', { 'anonymize_ip': true});
      • 13 Interpreting Time Series with CCDC
        • -
      • 14 Data Fusion: Merging Classification Streams -
        • -
      • 15 Exploring Lagged Effects in Time Series  +
      • 15 Exploring Lagged Effects in Time Series
          -
        • 15.1 Autocovariance and Autocorrelation
          • +
        • 15.1 Autocovariance and Autocorrelation
        • 15.2 Cross-Covariance and Cross-Correlation
        • 15.3 Auto-Regressive Models
        • Synthesis
          • @@ -351,10 +406,19 @@ gtag('config', 'G-RK9ZLZQ6GL', { 'anonymize_ip': true}); -

          One of the paradigm-changing features of Earth Engine is the ability to access decades of imagery without the previous limitation of needing to download all the data to a local disk for processing. Because remote-sensing data files can be enormous, this used to limit many projects to viewing two or three images from different periods. With Earth Engine, users can access tens or hundreds of thousands of images to understand the status of places across decades.

          +

          One of the paradigm-changing features of Earth Engine is the ability to access decades of imagery without the previous limitation of needing to download all the data to a local disk for processing. Because remote-sensing data files can be enormous, this used to limit many projects to viewing two or three images from different periods. With Earth Engine, users can access tens or hundreds of thousands of images to understand the status of places across decades.

          6 Filter, Map, Reduce

          -

          ::: {.callout-tip} # Chapter Information

          +
          +
          +
          + +
          +
          +Chapter Information +
          +
          +

          Author

          Jeffrey A. Cardille

          @@ -366,7 +430,7 @@ gtag('config', 'G-RK9ZLZQ6GL', { 'anonymize_ip': true});

          Learning Outcomes

            -
          • Visualizing the concepts of filtering, mapping, and reducing with a hypothetical, non-programming example.
            • +
          • Visualizing the concepts of filtering, mapping, and reducing with a hypothetical, non-programming example.
          • Gaining context and experience with filtering an ImageCollection.
          • Learning how to efficiently map a user-written function over the images of a filtered ImageCollection.
          • Learning how to summarize a set of assembled values using Earth Engine reducers.
            • @@ -379,73 +443,94 @@ gtag('config', 'G-RK9ZLZQ6GL', { 'anonymize_ip': true});
          • Perform basic image analysis: select bands, compute indices, create masks (Part F2).
          +
          +

          Introduction

          Prior chapters focused on exploring individual images—for example, viewing the characteristics of single satellite images by displaying different combinations of bands (Chap. F1.1), viewing single images from different datasets (Chap. F1.2, Chap. F1.3), and exploring image processing principles (Parts F2, F3) as they are implemented for cloud-based remote sensing in Earth Engine. Each image encountered in those chapters was pulled from a larger assemblage of images taken from the same sensor. The chapters used a few ways to narrow down the number of images in order to view just one for inspection (Part F1) or manipulation (Part F2, Part F3).

          In this chapter and most of the chapters that follow, we will move from the domain of single images to the more complex and distinctive world of working with image collections, one of the fundamental data types within Earth Engine. The ability to conceptualize and manipulate entire image collections distinguishes Earth Engine and gives it considerable power for interpreting change and stability across space and time.

          -

          When looking for change or seeking to understand differences in an area through time, we often proceed through three ordered stages, which we will color code in this first explanatory part of the lab:

          +

          When looking for change or seeking to understand differences in an area through time, we often proceed through three ordered stages, which we will color code in this first explanatory part of the lab:

          1. Filter: selecting subsets of images based on criteria of interest.
          2. Map: manipulating each image in a set in some way to suit our goals. and
          3. Reduce: estimating characteristics of the time series.

          For users of other programming languages—R, MATLAB, C, Karel, and many others—this approach might seem awkward at first. We explain it below with a non-programming example: going to the store to buy milk.

          -

          Suppose you need to go shopping for milk, and you have two criteria for determining where you will buy your milk: location and price. The store needs to be close to your home, and as a first step in deciding whether to buy milk today, you want to identify the lowest price among those stores. You don’t know the cost of milk at any store ahead of time, so you need to efficiently contact each one and determine the minimum price to know whether it fits in your budget. If we were discussing this with a friend, we might say, “I need to find out how much milk costs at all the stores around here.” To solve that problem in a programming language, these words imply precise operations on sets of information. We can write the following “pseudocode,” which uses words that indicate logical thinking but that cannot be pasted directly into a program:

          +

          Suppose you need to go shopping for milk, and you have two criteria for determining where you will buy your milk: location and price. The store needs to be close to your home, and as a first step in deciding whether to buy milk today, you want to identify the lowest price among those stores. You don’t know the cost of milk at any store ahead of time, so you need to efficiently contact each one and determine the minimum price to know whether it fits in your budget. If we were discussing this with a friend, we might say, “I need to find out how much milk costs at all the stores around here.” To solve that problem in a programming language, these words imply precise operations on sets of information. We can write the following “pseudocode,” which uses words that indicate logical thinking but that cannot be pasted directly into a program:

          AllStoresOnEarth.filterNearbyStores.filterStoresWithMilk.getMilkPricesFromEachStore.determineTheMinimumValue

          -

          Imagine doing these actions not on a computer but in a more old-fashioned way: calling on the telephone for milk prices, writing the milk prices on paper, and inspecting the list to find the lowest value. In this approach, we begin with AllStoresOnEarth, since there is at least some possibility that we could decide to visit any store on Earth, a set that could include millions of stores, with prices for millions or billions of items. A wise first action would be to limit ourselves to nearby stores. Asking to filterNearbyStores would reduce the number of potential stores to hundreds, depending on how far we are willing to travel for milk. Then, working with that smaller set, we further filterStoresWithMilk, limiting ourselves to stores that sell our target item. At that point in the filtering, imagine that just 10 possibilities remain. Then, by telephone, we getMilkPricesFromEachStore, making a short paper list of prices. We then scan the list to determineTheMinimumValue to decide which store to visit.

          -

          In that example, each color plays a different role in the workflow. The AllStoresOnEarth set, any one of which might contain inexpensive milk, is an enormous collection. The filtering actions filterNearbyStores and filterStoresWithMilk are operations that can happen on any set of stores. These actions take a set of stores, do some operation to limit that set, and return that smaller set of stores as an answer. The action to getMilkPricesFromEachStore takes a simple idea—calling a store for a milk price—and “maps” it over a given set of stores. Finally, with the list of nearby milk prices assembled, the action to determineTheMinimumValue, a general idea that could be applied to any list of numbers, identifies the cheapest one.

          -

          The list of steps above might seem almost too obvious, but the choice and order of operations can have a big impact on the feasibility of the problem. Imagine if we had decided to do the same operations in a slightly different order:

          +

          Imagine doing these actions not on a computer but in a more old-fashioned way: calling on the telephone for milk prices, writing the milk prices on paper, and inspecting the list to find the lowest value. In this approach, we begin with AllStoresOnEarth, since there is at least some possibility that we could decide to visit any store on Earth, a set that could include millions of stores, with prices for millions or billions of items. A wise first action would be to limit ourselves to nearby stores. Asking to filterNearbyStores would reduce the number of potential stores to hundreds, depending on how far we are willing to travel for milk. Then, working with that smaller set, we further filterStoresWithMilk, limiting ourselves to stores that sell our target item. At that point in the filtering, imagine that just 10 possibilities remain. Then, by telephone, we getMilkPricesFromEachStore, making a short paper list of prices. We then scan the list to determineTheMinimumValue to decide which store to visit.

          +

          In that example, each color plays a different role in the workflow. The AllStoresOnEarth set, any one of which might contain inexpensive milk, is an enormous collection. The filtering actions filterNearbyStores and filterStoresWithMilk are operations that can happen on any set of stores. These actions take a set of stores, do some operation to limit that set, and return that smaller set of stores as an answer. The action to getMilkPricesFromEachStore takes a simple idea—calling a store for a milk price—and “maps” it over a given set of stores. Finally, with the list of nearby milk prices assembled, the action to determineTheMinimumValue, a general idea that could be applied to any list of numbers, identifies the cheapest one.

          +

          The list of steps above might seem almost too obvious, but the choice and order of operations can have a big impact on the feasibility of the problem. Imagine if we had decided to do the same operations in a slightly different order:

          AllStoresOnEarth.filterStoresWithMilk.getMilkPricesFromEachStore.filterNearbyStores.determineMinimumValue

          -

          In this approach, we first identify all the stores on Earth that have milk, then contact them one by one to get their current milk price. If the contact is done by phone, this could be a painfully slow process involving millions of phone calls. It would take considerable “processing” time to make each call, and careful work to record each price onto a giant list. Processing the operations in this order would demand that only after entirely finishing the process of contacting every milk proprietor on Earth, we then identify the ones on our list that are not nearby enough to visit, then scan the prices on the list of nearby stores to find the cheapest one. This should ultimately give the same answer as the more efficient first example, but only after requiring so much effort that we might want to give up.

          +

          In this approach, we first identify all the stores on Earth that have milk, then contact them one by one to get their current milk price. If the contact is done by phone, this could be a painfully slow process involving millions of phone calls. It would take considerable “processing” time to make each call, and careful work to record each price onto a giant list. Processing the operations in this order would demand that only after entirely finishing the process of contacting every milk proprietor on Earth, we then identify the ones on our list that are not nearby enough to visit, then scan the prices on the list of nearby stores to find the cheapest one. This should ultimately give the same answer as the more efficient first example, but only after requiring so much effort that we might want to give up.

          In addition to the greater order of magnitude of the list size, you can see that there are also possible slow points in the process. Could you make a million phone calls yourself? Maybe, but it might be pretty appealing to hire, say, 1000 people to help. While being able to make a large number of calls in parallel would speed up the calling stage, it’s important to note that you would need to wait for all 1000 callers to return their sublists of prices. Why wait? Nearby stores could be on any caller’s sublist, so any caller might be the one to find the lowest nearby price. The identification of the lowest nearby price would need to wait for the slowest caller, even if it turned out that all of that last caller’s prices came from stores on the other side of the world.

          This counterexample would also have other complications—such as the need to track store locations on the list of milk prices—that could present serious problems if you did those operations in that unwise order. For now, the point is to filter, then map, then reduce. Below, we’ll apply these concepts to image collections.

          6.1 Filtering Image Collections in Earth Engine

          -

          The first part of the filter, map, reduce paradigm is “filtering” to get a smaller ImageCollection from a larger one. As in the milk example, filters take a large set of items, limit it by some criterion, and return a smaller set for consideration. Here, filters take an ImageCollection, limit it by some criterion of date, location, or image characteristics, and return a smaller ImageCollection (Fig. F4.0.1).

          -

          -

          Fig. 4.0.1 Filter, map, reduce as applied to image collections in Earth Engine

          -

          As described first in Chap. F1.2, the Earth Engine API provides a set of filters for the ImageCollection type. The filters can limit an ImageCollection based on spatial, temporal, or attribute characteristics. Filters were used in Parts F1, F2, and F3 without much context or explanation, to isolate an image from an ImageCollection for inspection or manipulation. The information below should give perspective on that work while introducing some new tools for filtering image collections.

          -

          Below are three examples of limiting a Landsat 5 ImageCollection by characteristics and assessing the size of the resulting set.

          -

          FilterDate This takes an ImageCollection as input and returns an ImageCollection whose members satisfy the specified date criteria. We’ll adapt the earlier filtering logic seen in Chap. F1.2:

          -

          var imgCol = ee.ImageCollection(‘LANDSAT/LT05/C02/T1_L2’);
          -// How many Tier 1 Landsat 5 images have ever been collected?
          -print(“All images ever:”, imgCol.size()); // A very large number

          -

          // How many images were collected in the 2000s?
          -var startDate = ‘2000-01-01’;
          -var endDate = ‘2010-01-01’;

          -

          var imgColfilteredByDate = imgCol.filterDate(startDate, endDate);
          -print(“All images 2000-2010:”, imgColfilteredByDate.size());
          -// A smaller (but still large) number

          -

          After running the code, you should get a very large number for the full set of images. You also will likely get a very large number for the subset of images over the decade-scale interval.  

          -

          FilterBounds It may be that—similar to the milk example—only images near to a place of interest are useful for you. As first presented in Part F1, filterBounds takes an ImageCollection as input and returns an ImageCollection whose images surround a specified location. If we take the ImageCollection that was filtered by date and then filter it by bounds, we will have filtered the collection to those images near a specified point within the specified date interval. With the code below, we’ll count the number of images in the Shanghai vicinity, first visited in Chap. F1.1, from the early 2000s:

          -

          var ShanghaiImage = ee.Image(   ‘LANDSAT/LT05/C02/T1_L2/LT05_118038_20000606’);
          -Map.centerObject(ShanghaiImage, 9);

          -

          var imgColfilteredByDateHere = imgColfilteredByDate.filterBounds(Map   .getCenter());
          -print(“All images here, 2000-2010:”, imgColfilteredByDateHere
          -.size()); // A smaller number

          -

          If you’d like, you could take a few minutes to explore the behavior of the script in different parts of the world. To do that, you would need to comment out the Map.centerObject command to keep the map from moving to that location each time you run the script.

          -

          Filter by Other Image Metadata        As first explained in Chap. F1.3, the date and location of an image are characteristics stored with each image. Another important factor in image processing is the cloud cover, an image-level value computed for each image in many collections, including the Landsat and Sentinel-2 collections. The overall cloudiness score might be stored under different metadata tag names in different data sets. For example, for Sentinel-2, this overall cloudiness score is stored in the CLOUDY_PIXEL_PERCENTAGE metadata field. For Landsat 5, the ImageCollection we are using in this example, the image-level cloudiness score is stored using the tag CLOUD_COVER. If you are unfamiliar with how to find this information, these skills are first presented in Part F1.

          -

          Here, we will access the ImageCollection that we just built using filterBounds and filterDate, and then further filter the images by the image-level cloud cover score, using the filterMetadata function.

          -

          Next, let’s remove any images with 50% or more cloudiness. As will be described in subsequent chapters working with per-pixel cloudiness information, you might want to retain those images in a real-life study, if you feel some values within cloudy images might be useful. For now, to illustrate the filtering concept, let’s keep only images whose image-level cloudiness values indicate that the cloud coverage is lower than 50%. Here, we will take the set already filtered by bounds and date, and further filter it using the cloud percentage into a new ImageCollection. Add this line to the script to filter by cloudiness and print the size to the Console.

          -

          var L5FilteredLowCloudImages = imgColfilteredByDateHere
          -   .filterMetadata(‘CLOUD_COVER’, ‘less_than’, 50);
          -print(“Less than 50% clouds in this area, 2000-2010”,
          -   L5FilteredLowCloudImages.size()); // A smaller number

          -

          Filtering in an Efficient Order As you saw earlier in the hypothetical milk example, we typically filter, then map, and then reduce, in that order. In the same way that we would not want to call every store on Earth, preferring instead to narrow down the list of potential stores first, we filter images first in our workflow in Earth Engine. In addition, you may have noticed that the ordering of the filters within the filtering stage also mattered in the milk example. This is also true in Earth Engine. For problems with a non-global spatial component in which filterBounds is to be used, it is most efficient to do that spatial filtering first.

          -

          In the code below, you will see that you can “chain” the filter commands, which are then executed from left to right. Below, we chain the filters in the same order as you specified above. Note that it gives an ImageCollection of the same size as when you applied the filters one at a time.

          -

          var chainedFilteredSet = imgCol.filterDate(startDate, endDate)
          -   .filterBounds(Map.getCenter())
          -   .filterMetadata(‘CLOUD_COVER’, ‘less_than’, 50);
          -print(‘Chained: Less than 50% clouds in this area, 2000-2010’,
          -   chainedFilteredSet.size());

          -

          In the code below, we chain the filters in a more efficient order, implementing filterBounds first. This, too, gives an ImageCollection of the same size as when you applied the filters in the less efficient order, whether the filters were chained or not.

          -

          var efficientFilteredSet = imgCol.filterBounds(Map.getCenter())
          -   .filterDate(startDate, endDate)
          -   .filterMetadata(‘CLOUD_COVER’, ‘less_than’, 50);
          +

          The first part of the filter, map, reduce paradigm is “filtering” to get a smaller ImageCollection from a larger one. As in the milk example, filters take a large set of items, limit it by some criterion, and return a smaller set for consideration. Here, filters take an ImageCollection, limit it by some criterion of date, location, or image characteristics, and return a smaller ImageCollection (Fig. F4.0.1).

          +
          +
          +

          +

          Fig. 4.0.1 Filter, map, reduce as applied to image collections in Earth Engine

          +
          +
          +

          As described first in Chap. F1.2, the Earth Engine API provides a set of filters for the ImageCollection type. The filters can limit an ImageCollection based on spatial, temporal, or attribute characteristics. Filters were used in Parts F1, F2, and F3 without much context or explanation, to isolate an image from an ImageCollection for inspection or manipulation. The information below should give perspective on that work while introducing some new tools for filtering image collections.

          +

          Below are three examples of limiting a Landsat 5 ImageCollection by characteristics and assessing the size of the resulting set.

          +

          FilterDate This takes an ImageCollection as input and returns an ImageCollection whose members satisfy the specified date criteria. We’ll adapt the earlier filtering logic seen in Chap. F1.2:

          +

          var imgCol = ee.ImageCollection(‘LANDSAT/LT05/C02/T1_L2’);

          +
          // How many Tier 1 Landsat 5 images have ever been collected?  
+print("All images ever: ", imgCol.size()); // A very large number  
+  
+// How many images were collected in the 2000s?  
+var startDate = '2000-01-01';  
+var endDate = '2010-01-01';  
+  
+var imgColfilteredByDate = imgCol.filterDate(startDate, endDate);  
+print("All images 2000-2010: ", imgColfilteredByDate.size());  
+// A smaller (but still large) number
+
+After running the code, you should get a very large number for the full set of images. You also will likely get a very large number for the subset of images over the decade-scale interval.  
+
+FilterBounds It may be that—similar to the milk example—only images near to a place of interest are useful for you. As first presented in Part F1, filterBounds takes an ImageCollection as input and returns an ImageCollection whose images surround a specified location. If we take the ImageCollection that was filtered by date and then filter it by bounds, we will have filtered the collection to those images near a specified point within the specified date interval. With the code below, we’ll count the number of images in the Shanghai vicinity, first visited in Chap. F1.1, from the early 2000s:
+
+var ShanghaiImage = ee.Image(   'LANDSAT/LT05/C02/T1_L2/LT05_118038_20000606');  
+Map.centerObject(ShanghaiImage, 9);  
+  
+var imgColfilteredByDateHere = imgColfilteredByDate.filterBounds(Map   .getCenter());  
+print("All images here, 2000-2010: ", imgColfilteredByDateHere  
+.size()); // A smaller number
+
+If you’d like, you could take a few minutes to explore the behavior of the script in different parts of the world. To do that, you would need to comment out the Map.centerObject command to keep the map from moving to that location each time you run the script.
+
+  
+Filter by Other Image Metadata        As first explained in Chap. F1.3, the date and location of an image are characteristics stored with each image. Another important factor in image processing is the cloud cover, an image-level value computed for each image in many collections, including the Landsat and Sentinel-2 collections. The overall cloudiness score might be stored under different metadata tag names in different data sets. For example, for Sentinel-2, this overall cloudiness score is stored in the CLOUDY_PIXEL_PERCENTAGE metadata field. For Landsat 5, the ImageCollection we are using in this example, the image-level cloudiness score is stored using the tag CLOUD_COVER. If you are unfamiliar with how to find this information, these skills are first presented in Part F1.
+
+Here, we will access the ImageCollection that we just built using filterBounds and filterDate, and then further filter the images by the image-level cloud cover score, using the filterMetadata function.
+
+Next, let’s remove any images with 50% or more cloudiness. As will be described in subsequent chapters working with per-pixel cloudiness information, you might want to retain those images in a real-life study, if you feel some values within cloudy images might be useful. For now, to illustrate the filtering concept, let’s keep only images whose image-level cloudiness values indicate that the cloud coverage is lower than 50%. Here, we will take the set already filtered by bounds and date, and further filter it using the cloud percentage into a new ImageCollection. Add this line to the script to filter by cloudiness and print the size to the Console.
+
+var L5FilteredLowCloudImages = imgColfilteredByDateHere  
+   .filterMetadata('CLOUD_COVER', 'less_than', 50);  
+print("Less than 50% clouds in this area, 2000-2010",  
+   L5FilteredLowCloudImages.size()); // A smaller number
+
+Filtering in an Efficient Order As you saw earlier in the hypothetical milk example, we typically filter, then map, and then reduce, in that order. In the same way that we would not want to call every store on Earth, preferring instead to narrow down the list of potential stores first, we filter images first in our workflow in Earth Engine. In addition, you may have noticed that the ordering of the filters within the filtering stage also mattered in the milk example. This is also true in Earth Engine. For problems with a non-global spatial component in which filterBounds is to be used, it is most efficient to do that spatial filtering first.
+
+In the code below, you will see that you can “chain” the filter commands, which are then executed from left to right. Below, we chain the filters in the same order as you specified above. Note that it gives an ImageCollection of the same size as when you applied the filters one at a time.
+
+var chainedFilteredSet = imgCol.filterDate(startDate, endDate)  
+   .filterBounds(Map.getCenter())  
+   .filterMetadata('CLOUD_COVER', 'less_than', 50);  
+print('Chained: Less than 50% clouds in this area, 2000-2010',  
+   chainedFilteredSet.size());
          +

          In the code below, we chain the filters in a more efficient order, implementing filterBounds first. This, too, gives an ImageCollection of the same size as when you applied the filters in the less efficient order, whether the filters were chained or not.

          +

          var efficientFilteredSet = imgCol.filterBounds(Map.getCenter())
          +.filterDate(startDate, endDate)
          +.filterMetadata(‘CLOUD_COVER’, ‘less_than’, 50);
          print(‘Efficient filtering: Less than 50% clouds in this area, 2000-2010’,
          -   efficientFilteredSet.size());

          -

          Each of the two chained sets of operations will give the same result as before for the number of images. While the second order is more efficient, both approaches are likely to return the answer to the Code Editor at roughly the same time for this very small example. The order of operations is most important in larger problems in which you might be challenged to manage memory carefully. As in the milk example in which you narrowed geographically first, it is good practice in Earth Engine to order the filters with the filterBounds first, followed by metadata filters in order of decreasing specificity.

          +efficientFilteredSet.size());

          +

          Each of the two chained sets of operations will give the same result as before for the number of images. While the second order is more efficient, both approaches are likely to return the answer to the Code Editor at roughly the same time for this very small example. The order of operations is most important in larger problems in which you might be challenged to manage memory carefully. As in the milk example in which you narrowed geographically first, it is good practice in Earth Engine to order the filters with the filterBounds first, followed by metadata filters in order of decreasing specificity.

          @@ -456,28 +541,28 @@ Note
          -

          Code Checkpoint F40a. The book’s repository contains a script that shows what your code should look like at this point.

          +

          Code Checkpoint F40a. The book’s repository contains a script that shows what your code should look like at this point.

          Now, with an efficiently filtered collection that satisfies our chosen criteria, we will next explore the second stage: executing a function for all of the images in the set.

          6.2 Mapping over Image Collections in Earth Engine

          -

          In Chap. F3.1, we calculated the Enhanced Vegetation Index (EVI) in very small steps to illustrate band arithmetic on satellite images. In that chapter, code was called once, on a single image. What if we wanted to compute the EVI in the same way for every image of an entire ImageCollection? Here, we use the key tool for the second part of the workflow in Earth Engine, a .map command (Fig. F4.0.1). This is roughly analogous to the step of making phone calls in the milk example that began this chapter, in which you took a list of store names and transformed it through effort into a list of milk prices.

          -

          Before beginning to code the EVI functionality, it’s worth noting that the word “map” is encountered in multiple settings during cloud-based remote sensing, and it’s important to be able to distinguish the uses. A good way to think of it is that “map” can act as a verb or as a noun in Earth Engine. There are two uses of “map” as a noun. We might refer casually to “the map,” or more precisely to “the  Map panel”; these terms refer to the place where the images are shown in the code interface. A second way “map” is used as a noun is to refer to an Earth Engine object, which has functions that can be called on it. Examples of this are the familiar Map.addLayer and Map.setCenter. Where that use of the word is intended, it will be shown in purple text and capitalized in the Code Editor. What we are discussing here is the use of .map as a verb, representing the idea of performing a set of actions repeatedly on a set. This is typically referred to as “mapping over the set.”  

          -

          To map a given set of operations efficiently over an entire ImageCollection, the processing needs to be set up in a particular way. Users familiar with other programming languages might expect to see “loop” code to do this, but the processing is not done exactly that way in Earth Engine. Instead, we will create a function, and then map it over the ImageCollection. To begin, envision creating a function that takes exactly one parameter, an ee.Image. The function is then designed to perform a specified set of operations on the input ee.Image and then, importantly, returns an ee.Image as the last step of the function. When we map that function over an ImageCollection, as we’ll illustrate below, the effect is that we begin with an ImageCollection, do operations to each image, and receive a processed ImageCollection as the output.

          -

          What kinds of functions could we create? For example, you could imagine a function taking an image and returning an image whose pixels have the value 1 where the value of a given band was lower than a certain threshold, and 0 otherwise. The effect of mapping this function would be an entire ImageCollection of images with zeroes and ones representing the results of that test on each image. Or you could imagine a function computing a complex self-defined index and sending back an image of that index calculated in each pixel. Here, we’ll create a function to compute the EVI for any input Landsat 5 image and return the one-band image for which the index is computed for each pixel. Copy and paste the function definition below into the Code Editor, adding it to the end of the script from the previous section.

          -

          var makeLandsat5EVI = function(oneL5Image) {   // compute the EVI for any Landsat 5 image. Note it’s specific to   // Landsat 5 images due to the band numbers. Don’t run this exact   // function for images from sensors other than Landsat 5.   // Extract the bands and divide by 1e4 to account for scaling done.   var nirScaled = oneL5Image.select(‘SRvide(10000);   var redScaled = oneL5Image.select(’SR_B3’).divide(10000);   var blueScaled = oneL5Image.select(‘SR_B1’).divide(10000);   // Calculate the numerator, note that order goes from left to right.   var numeratorEVI = (nirScaled.subtract(redScaled)).multiply(       2.5);   // Calculate the denominator   var denomClause1 = redScaled.multiply(6);   var denomClause2 = blueScaled.multiply(7.5);   var denominatorEVI = nirScaled.add(denomClause1).subtract(
          -       denomClause2).add(1);   // Calculate EVI and name it.   var landsat5EVI = numeratorEVI.divide(denominatorEVI).rename(       ‘EVI’);   return (landsat5EVI);
          +

          In Chap. F3.1, we calculated the Enhanced Vegetation Index (EVI) in very small steps to illustrate band arithmetic on satellite images. In that chapter, code was called once, on a single image. What if we wanted to compute the EVI in the same way for every image of an entire ImageCollection? Here, we use the key tool for the second part of the workflow in Earth Engine, a .map command (Fig. F4.0.1). This is roughly analogous to the step of making phone calls in the milk example that began this chapter, in which you took a list of store names and transformed it through effort into a list of milk prices.

          +

          Before beginning to code the EVI functionality, it’s worth noting that the word “map” is encountered in multiple settings during cloud-based remote sensing, and it’s important to be able to distinguish the uses. A good way to think of it is that “map” can act as a verb or as a noun in Earth Engine. There are two uses of “map” as a noun. We might refer casually to “the map,” or more precisely to “the Map panel”; these terms refer to the place where the images are shown in the code interface. A second way “map” is used as a noun is to refer to an Earth Engine object, which has functions that can be called on it. Examples of this are the familiar Map.addLayer and Map.setCenter. Where that use of the word is intended, it will be shown in purple text and capitalized in the Code Editor. What we are discussing here is the use of .map as a verb, representing the idea of performing a set of actions repeatedly on a set. This is typically referred to as “mapping over the set.”

          +

          To map a given set of operations efficiently over an entire ImageCollection, the processing needs to be set up in a particular way. Users familiar with other programming languages might expect to see “loop” code to do this, but the processing is not done exactly that way in Earth Engine. Instead, we will create a function, and then map it over the ImageCollection. To begin, envision creating a function that takes exactly one parameter, an ee.Image. The function is then designed to perform a specified set of operations on the input ee.Image and then, importantly, returns an ee.Image as the last step of the function. When we map that function over an ImageCollection, as we’ll illustrate below, the effect is that we begin with an ImageCollection, do operations to each image, and receive a processed ImageCollection as the output.

          +

          What kinds of functions could we create? For example, you could imagine a function taking an image and returning an image whose pixels have the value 1 where the value of a given band was lower than a certain threshold, and 0 otherwise. The effect of mapping this function would be an entire ImageCollection of images with zeroes and ones representing the results of that test on each image. Or you could imagine a function computing a complex self-defined index and sending back an image of that index calculated in each pixel. Here, we’ll create a function to compute the EVI for any input Landsat 5 image and return the one-band image for which the index is computed for each pixel. Copy and paste the function definition below into the Code Editor, adding it to the end of the script from the previous section.

          +

          var makeLandsat5EVI = function(oneL5Image) { // compute the EVI for any Landsat 5 image. Note it’s specific to // Landsat 5 images due to the band numbers. Don’t run this exact // function for images from sensors other than Landsat 5. // Extract the bands and divide by 1e4 to account for scaling done. var nirScaled = oneL5Image.select(‘SRvide(10000); var redScaled = oneL5Image.select(’SR_B3’).divide(10000); var blueScaled = oneL5Image.select(‘SR_B1’).divide(10000); // Calculate the numerator, note that order goes from left to right. var numeratorEVI = (nirScaled.subtract(redScaled)).multiply( 2.5); // Calculate the denominator var denomClause1 = redScaled.multiply(6); var denomClause2 = blueScaled.multiply(7.5); var denominatorEVI = nirScaled.add(denomClause1).subtract(
          +denomClause2).add(1); // Calculate EVI and name it. var landsat5EVI = numeratorEVI.divide(denominatorEVI).rename( ‘EVI’); return (landsat5EVI);
          };

          -

          It is worth emphasizing that, in general, band names are specific to each ImageCollection. As a result, if that function were run on an image without the band ‘SR_B4’, for example, the function call would fail. Here, we have emphasized in the function’s name that it is specifically for creating EVI for Landsat 5.  

          -

          The function makeLandsat5EVI is built to receive a single image, select the proper bands for calculating EVI, make the calculation, and return a one-banded image. If we had the name of each image comprising our ImageCollection, we could enter the names into the Code Editor and call the function one at a time for each, assembling the images into variables, and then combining them into an ImageCollection. This would be very tedious and highly prone to mistakes: lists of items might get mistyped, an image might be missed, etc. Instead, as mentioned above, we will use .map. With the code below, let’s print the information about the cloud-filtered collection and display it, execute the .map command, and explore the resulting ImageCollection.

          -

          var L5EVIimages = efficientFilteredSet.map(makeLandsat5EVI);
          +

          It is worth emphasizing that, in general, band names are specific to each ImageCollection. As a result, if that function were run on an image without the band ‘SR_B4’, for example, the function call would fail. Here, we have emphasized in the function’s name that it is specifically for creating EVI for Landsat 5.

          +

          The function makeLandsat5EVI is built to receive a single image, select the proper bands for calculating EVI, make the calculation, and return a one-banded image. If we had the name of each image comprising our ImageCollection, we could enter the names into the Code Editor and call the function one at a time for each, assembling the images into variables, and then combining them into an ImageCollection. This would be very tedious and highly prone to mistakes: lists of items might get mistyped, an image might be missed, etc. Instead, as mentioned above, we will use .map. With the code below, let’s print the information about the cloud-filtered collection and display it, execute the .map command, and explore the resulting ImageCollection.

          +

          var L5EVIimages = efficientFilteredSet.map(makeLandsat5EVI);
          print(‘Verifying that the .map gives back the same number of images:’,
          -   L5EVIimages.size());
          +L5EVIimages.size());
          print(L5EVIimages);

          Map.addLayer(L5EVIimages, {}, ‘L5EVIimages’, 1, 1);

          -

          After entering and executing this code, you will see a grayscale image. If you look closely at the edges of the image, you might spot other images drawn behind it in a way that looks somewhat like a stack of papers on a table. This is the drawing of the ImageCollection made from the makeLandsat5EVI function. You can select the Inspector panel and click on one of the grayscale pixels to view the values of the entire ImageCollection. After clicking on a pixel, look for the Series tag by opening and closing the list of items. When you open that tag, you will see a chart of the EVI values at that pixel, created by mapping the makeLandsat5EVI function over the filtered ImageCollection.

          +

          After entering and executing this code, you will see a grayscale image. If you look closely at the edges of the image, you might spot other images drawn behind it in a way that looks somewhat like a stack of papers on a table. This is the drawing of the ImageCollection made from the makeLandsat5EVI function. You can select the Inspector panel and click on one of the grayscale pixels to view the values of the entire ImageCollection. After clicking on a pixel, look for the Series tag by opening and closing the list of items. When you open that tag, you will see a chart of the EVI values at that pixel, created by mapping the makeLandsat5EVI function over the filtered ImageCollection.

          @@ -488,34 +573,38 @@ Note
          -

          Code Checkpoint F40b. The book’s repository contains a script that shows what your code should look like at this point.

          +

          Code Checkpoint F40b. The book’s repository contains a script that shows what your code should look like at this point.

          6.3 Reducing an Image Collection

          -

          The third part of the filter, map, reduce paradigm is “reducing” values in an ImageCollection to extract meaningful values (Fig. F4.0.1). In the milk example, we reduced a large list of milk prices to find the minimum value. The Earth Engine API provides a large set of reducers for reducing a set of values to a summary statistic.

          -

          Here, you can think of each location, after the calculation of EVI has been executed though the .map command, as having a list of EVI values on it. Each pixel contains a potentially very large set of EVI values; the stack might be 15 items high in one location and perhaps 200, 2000, or 200,000 items high in another location, especially if a looser set of filters had been used.

          -

          The code below computes the mean value, at every pixel, of the ImageCollection L5EVIimages created above. Add it at the bottom of your code.

          -

          var L5EVImean = L5EVIimages.reduce(ee.Reducer.mean());
          +

          The third part of the filter, map, reduce paradigm is “reducing” values in an ImageCollection to extract meaningful values (Fig. F4.0.1). In the milk example, we reduced a large list of milk prices to find the minimum value. The Earth Engine API provides a large set of reducers for reducing a set of values to a summary statistic.

          +

          Here, you can think of each location, after the calculation of EVI has been executed though the .map command, as having a list of EVI values on it. Each pixel contains a potentially very large set of EVI values; the stack might be 15 items high in one location and perhaps 200, 2000, or 200,000 items high in another location, especially if a looser set of filters had been used.

          +

          The code below computes the mean value, at every pixel, of the ImageCollection L5EVIimages created above. Add it at the bottom of your code.

          +

          var L5EVImean = L5EVIimages.reduce(ee.Reducer.mean());
          print(L5EVImean);
          Map.addLayer(L5EVImean, {
          -   min: -1,
          -   max: 2,
          -   palette: [‘red’, ‘white’, ‘green’]
          +min: -1,
          +max: 2,
          +palette: [‘red’, ‘white’, ‘green’]
          }, ‘Mean EVI’);

          -

          Using the same principle, the code below computes and draws the median value of the ImageCollection in every pixel.

          -

          var L5EVImedian = L5EVIimages.reduce(ee.Reducer.median());
          +

          Using the same principle, the code below computes and draws the median value of the ImageCollection in every pixel.

          +

          var L5EVImedian = L5EVIimages.reduce(ee.Reducer.median());
          print(L5EVImedian);
          Map.addLayer(L5EVImedian, {
          -   min: -1,
          -   max: 2,
          -   palette: [‘red’, ‘white’, ‘green’]
          +min: -1,
          +max: 2,
          +palette: [‘red’, ‘white’, ‘green’]
          }, ‘Median EVI’);

          -

          -

          Fig. 4.0.2 The effects of two reducers on mapped EVI values in a filtered ImageCollection: mean image (above), and median image (below)

          -

          There are many more reducers that work with an ImageCollection to produce a wide range of summary statistics. Reducers are not limited to returning only one item from the reduction. The minMax reducer, for example, returns a two-band image for each band it is given, one for the minimum and one for the maximum.

          +
          +
          +

          +

          Fig. 4.0.2 The effects of two reducers on mapped EVI values in a filtered ImageCollection: mean image (above), and median image (below)

          +
          +
          +

          There are many more reducers that work with an ImageCollection to produce a wide range of summary statistics. Reducers are not limited to returning only one item from the reduction. The minMax reducer, for example, returns a two-band image for each band it is given, one for the minimum and one for the maximum.

          The reducers described here treat each pixel independently. In subsequent chapters in Part F4, you will see other kinds of reducers—for example, ones that summarize the characteristics in the neighborhood surrounding each pixel.

          @@ -527,27 +616,27 @@ Note
          -

          Code Checkpoint F40c. The book’s repository contains a script that shows what your code should look like at this point.

          +

          Code Checkpoint F40c. The book’s repository contains a script that shows what your code should look like at this point.

          Synthesis

          -

          Assignment 1. Compare the mean and median images produced in Sect. 3 (Fig. 4.0.2). In what ways do they look different, and in what ways do they look alike? To understand how they work, pick a pixel and inspect the EVI values computed. In your opinion, which is a better representative of the data set?

          +

          Assignment 1. Compare the mean and median images produced in Sect. 3 (Fig. 4.0.2). In what ways do they look different, and in what ways do they look alike? To understand how they work, pick a pixel and inspect the EVI values computed. In your opinion, which is a better representative of the data set?

          Assignment 2. Adjust the filters to filter a different proportion of clouds, or a different date range. What effects do these changes have on the number of images and the look of the reductions made from them?

          -

          Assignment 3. Explore the ee.Filter options in the API documentation, and select a different filter that might be of interest. Filter images using it, and comment on the number of images and the reductions made from them.

          -

          Assignment 4. Change the EVI function so that it returns the original image with the EVI band appended by replacing the return statement with this: return (oneL5Image.addBands(landsat5EVI))

          -

          What does the median reducer return in that case? Some EVI values are 0. What are the conditions in which this occurs?  

          -

          Assignment 5. Choose a date and location that is important to you (e.g., your birthday and your place of birth). Filter Landsat imagery to get all the low-cloud imagery at your location within 6 months of the date. Then, reduce the ImageCollection to find the median EVI. Describe the image and how representative of the full range of values it is, in your opinion.

          +

          Assignment 3. Explore the ee.Filter options in the API documentation, and select a different filter that might be of interest. Filter images using it, and comment on the number of images and the reductions made from them.

          +

          Assignment 4. Change the EVI function so that it returns the original image with the EVI band appended by replacing the return statement with this: return (oneL5Image.addBands(landsat5EVI))

          +

          What does the median reducer return in that case? Some EVI values are 0. What are the conditions in which this occurs?

          +

          Assignment 5. Choose a date and location that is important to you (e.g., your birthday and your place of birth). Filter Landsat imagery to get all the low-cloud imagery at your location within 6 months of the date. Then, reduce the ImageCollection to find the median EVI. Describe the image and how representative of the full range of values it is, in your opinion.

          Conclusion

          -

          In this chapter, you learned about the paradigm of filter, map, reduce. You learned how to use these tools to sift through, operate on, and summarize a large set of images to suit your purposes. Using the Filter functionality, you learned how to take a large ImageCollection and filter away images that do not meet your criteria, retaining only those images that match a given set of characteristics. Using the Map functionality, you learned how to apply a function to each image in an ImageCollection, treating each image one at a time and executing a requested set of operations on each. Using the Reduce functionality, you learned how to summarize the elements of an ImageCollection, extracting summary values of interest. In the subsequent chapters of Part 4, you will encounter these concepts repeatedly, manipulating image collections according to your project needs using the building blocks seen here. By building on what you have done in this chapter, you will grow in your ability to do sophisticated projects in Earth Engine.

          +

          In this chapter, you learned about the paradigm of filter, map, reduce. You learned how to use these tools to sift through, operate on, and summarize a large set of images to suit your purposes. Using the Filter functionality, you learned how to take a large ImageCollection and filter away images that do not meet your criteria, retaining only those images that match a given set of characteristics. Using the Map functionality, you learned how to apply a function to each image in an ImageCollection, treating each image one at a time and executing a requested set of operations on each. Using the Reduce functionality, you learned how to summarize the elements of an ImageCollection, extracting summary values of interest. In the subsequent chapters of Part 4, you will encounter these concepts repeatedly, manipulating image collections according to your project needs using the building blocks seen here. By building on what you have done in this chapter, you will grow in your ability to do sophisticated projects in Earth Engine.

          7 Exploring Image Collections

          -

          ::: {.callout-tip} # Chapter Information

          +

          :::{.callout-tip} # Chapter Information

          Author

          Gennadii Donchyts

          @@ -558,10 +647,9 @@ Note

          Learning Outcomes

          -

           

            -
          • Inspecting the spatiotemporal extent and resolution of image collections by mapping image geometry and plotting image time properties.
            • -
          • Exploring properties of images stored in an ImageCollection by plotting charts and deriving statistics.
            • +
          • Inspecting the spatiotemporal extent and resolution of image collections by mapping image geometry and plotting image time properties.
            • +
          • Exploring properties of images stored in an ImageCollection by plotting charts and deriving statistics.
          • Filtering image collections by using stored or computed image properties.
          • Exploring the distribution of values stored in image pixels of an ImageCollection through percentile reducers.
          @@ -571,40 +659,48 @@ Note
          • Import images and image collections, filter, and visualize (Part F1).
          • Perform basic image analysis: select bands, compute indices, create masks (Part F2).
            • -
          • Summarize an ImageCollection with reducers (Chap. F4.0).
            • +
          • Summarize an ImageCollection with reducers (Chap. F4.0).
          -

          In the previous chapter (Chap. F4.0), the filter, map, reduce paradigm was introduced. The main goal of this chapter is to demonstrate some of the ways that those concepts can be used within Earth Engine to better understand the variability of values stored in image collections. Sect. 1 demonstrates how time-dependent values stored in the images of an ImageCollection can be inspected using the Code Editor user interface after filtering them to a limited spatiotemporal range (i.e., geometry and time ranges). Sect. 2 shows how the extent of images, as well as basic statistics, such as the number of observations, can be visualized to better understand the spatiotemporal extent of image collections. Then, Sects. 3 and 4 demonstrate how simple reducers such as mean and median, and more advanced reducers such as percentiles, can be used to better understand how the values of a filtered ImageCollection are distributed.

          +

          In the previous chapter (Chap. F4.0), the filter, map, reduce paradigm was introduced. The main goal of this chapter is to demonstrate some of the ways that those concepts can be used within Earth Engine to better understand the variability of values stored in image collections. Sect. 1 demonstrates how time-dependent values stored in the images of an ImageCollection can be inspected using the Code Editor user interface after filtering them to a limited spatiotemporal range (i.e., geometry and time ranges). Sect. 2 shows how the extent of images, as well as basic statistics, such as the number of observations, can be visualized to better understand the spatiotemporal extent of image collections. Then, Sects. 3 and 4 demonstrate how simple reducers such as mean and median, and more advanced reducers such as percentiles, can be used to better understand how the values of a filtered ImageCollection are distributed.

          7.1 Filtering and Inspecting an Image Collection

          -

          We will focus on the area in and surrounding Lisbon, Portugal. Below, we will define a point, lisbonPoint, located in the city; access the very large Landsat ImageCollection and limit it to the year 2020 and to the images that contain Lisbon; and select bands 6, 5, and 4 from each of the images in the resulting filtered ImageCollection.

          -

          // Define a region of interest as a point in Lisbon, Portugal.
          -var lisbonPoint = ee.Geometry.Point(-9.179473, 38.763948);

          -

          // Center the map at that point.
          -Map.centerObject(lisbonPoint, 16);

          -

          // filter the large ImageCollection to be just images from 2020
          -// around Lisbon. From each image, select true-color bands to draw
          -var filteredIC = ee.ImageCollection(‘LANDSAT/LC08/C02/T1_TOA’)
          -   .filterDate(‘2020-01-01’, ‘2021-01-01’)
          -   .filterBounds(lisbonPoint)
          -   .select([‘B6’, ‘B5’, ‘B4’]);

          -

          // Add the filtered ImageCollection so that we can inspect values
          -// via the Inspector tool
          -Map.addLayer(filteredIC, {}, ‘TOA image collection’);

          -

          The three selected bands (which correspond to SWIR1, NIR, and Red) display a false-color image that accentuates differences between different land covers (e.g., concrete, vegetation) in Lisbon. With the Inspector tab highlighted (Fig. F4.1.1), clicking on a point will bring up the values of bands 6, 5, and 4 from each of the images. If you open the Series option, you’ll see the values through time. For the specified point and for all other points in Lisbon (since they are all enclosed in the same Landsat scene), there are 16 images gathered in 2020. By following one of the graphed lines (in blue, yellow, or red) with your finger, you should be able to count that many distinct values. Moving the mouse along the lines will show the specific values and the image dates.

          -

          -

          Fig. F4.1.1 Inspect values in an ImageCollection at a selected point by making use of the Inspector tool in the Code Editor

          -

          We can also show this kind of chart automatically by making use of the ui.Chart function of the Earth Engine API. The following code snippet should result in the same chart as we could observe in the Inspector tab, assuming the same pixel is clicked.

          -

          // Construct a chart using values queried from image collection.
          -var chart = ui.Chart.image.series({
          -   imageCollection: filteredIC,
          -   region: lisbonPoint,
          -   reducer: ee.Reducer.first(),
          -   scale: 10
          -});

          -

          // Show the chart in the Console.
          -print(chart);

          +

          We will focus on the area in and surrounding Lisbon, Portugal. Below, we will define a point, lisbonPoint, located in the city; access the very large Landsat ImageCollection and limit it to the year 2020 and to the images that contain Lisbon; and select bands 6, 5, and 4 from each of the images in the resulting filtered ImageCollection.

          +
          // Define a region of interest as a point in Lisbon, Portugal.  
+var lisbonPoint = ee.Geometry.Point(-9.179473, 38.763948);  
+  
+// Center the map at that point.  
+Map.centerObject(lisbonPoint, 16);  
+  
+// filter the large ImageCollection to be just images from 2020  
+// around Lisbon. From each image, select true-color bands to draw  
+var filteredIC = ee.ImageCollection('LANDSAT/LC08/C02/T1_TOA')  
+   .filterDate('2020-01-01', '2021-01-01')  
+   .filterBounds(lisbonPoint)  
+   .select(['B6', 'B5', 'B4']);  
+  
+// Add the filtered ImageCollection so that we can inspect values  
+// via the Inspector tool  
+Map.addLayer(filteredIC, {}, 'TOA image collection');
          +

          The three selected bands (which correspond to SWIR1, NIR, and Red) display a false-color image that accentuates differences between different land covers (e.g., concrete, vegetation) in Lisbon. With the Inspector tab highlighted (Fig. F4.1.1), clicking on a point will bring up the values of bands 6, 5, and 4 from each of the images. If you open the Series option, you’ll see the values through time. For the specified point and for all other points in Lisbon (since they are all enclosed in the same Landsat scene), there are 16 images gathered in 2020. By following one of the graphed lines (in blue, yellow, or red) with your finger, you should be able to count that many distinct values. Moving the mouse along the lines will show the specific values and the image dates.

          +
          +
          +

          +

          Fig. F4.1.1 Inspect values in an ImageCollection at a selected point by making use of the Inspector tool in the Code Editor

          +
          +
          +

          We can also show this kind of chart automatically by making use of the ui.Chart function of the Earth Engine API. The following code snippet should result in the same chart as we could observe in the Inspector tab, assuming the same pixel is clicked.

          +
          // Construct a chart using values queried from image collection.  
+var chart = ui.Chart.image.series({  
+   imageCollection: filteredIC,  
+   region: lisbonPoint,  
+   reducer: ee.Reducer.first(),  
+   scale: 10  
+});  
+  
+// Show the chart in the Console.  
+print(chart);
          @@ -615,34 +711,43 @@ Note
          -

          Code Checkpoint F41a. The book’s repository contains a script that shows what your code should look like at this point.

          +

          Code Checkpoint F41a. The book’s repository contains a script that shows what your code should look like at this point.

          -

          ## How Many Images Are There, Everywhere on Earth?

          -

          Suppose we are interested to find out how many valid observations we have at every map pixel on Earth for a given ImageCollection. This enormously computationally demanding task is surprisingly easy to do in Earth Engine. The API provides a set of reducer functions to summarize values to a single number in each pixel, as described in Chap. F4.0. We can apply this reducer, count, to our filtered ImageCollection with the code below. We’ll return to the same data set and filter for 2020, but without the geographic limitation. This will assemble images from all over the world, and then count the number of images in each pixel. The following code does that count, and adds the resulting image to the map with a predefined red/yellow/green color palette stretched between values 0 and 50. Continue pasting the code below into the same script.

          -

          // compute and show the number of observations in an image collection
          -var count = ee.ImageCollection(‘LANDSAT/LC08/C02/T1_TOA’)
          -   .filterDate(‘2020-01-01’, ‘2021-01-01’)
          -   .select([‘B6’])
          -   .count();

          -

          // add white background and switch to HYBRID basemap
          -Map.addLayer(ee.Image(1), {
          -   palette: [‘white’]
          -}, ‘white’, true, 0.5);
          -Map.setOptions(‘HYBRID’);

          -

          // show image count
          -Map.addLayer(count, {
          -   min: 0,
          -   max: 50,
          -   palette: [‘d7191c’, ‘fdae61’, ‘ffffbf’, ‘a6d96a’,       ‘1a9641’]
          -}, ‘landsat 8 image count (2020)’);

          -

          // Center the map at that point.
          -Map.centerObject(lisbonPoint, 5);

          +
          +
          +

          7.2 How Many Images Are There, Everywhere on Earth?

          +

          Suppose we are interested to find out how many valid observations we have at every map pixel on Earth for a given ImageCollection. This enormously computationally demanding task is surprisingly easy to do in Earth Engine. The API provides a set of reducer functions to summarize values to a single number in each pixel, as described in Chap. F4.0. We can apply this reducer, count, to our filtered ImageCollection with the code below. We’ll return to the same data set and filter for 2020, but without the geographic limitation. This will assemble images from all over the world, and then count the number of images in each pixel. The following code does that count, and adds the resulting image to the map with a predefined red/yellow/green color palette stretched between values 0 and 50. Continue pasting the code below into the same script.

          +
          // compute and show the number of observations in an image collection  
+var count = ee.ImageCollection('LANDSAT/LC08/C02/T1_TOA')  
+   .filterDate('2020-01-01', '2021-01-01')  
+   .select(['B6'])  
+   .count();  
+  
+// add white background and switch to HYBRID basemap  
+Map.addLayer(ee.Image(1), {  
+   palette: ['white']  
+}, 'white', true, 0.5);  
+Map.setOptions('HYBRID');  
+  
+// show image count  
+Map.addLayer(count, {  
+   min: 0,  
+   max: 50,  
+   palette: ['d7191c', 'fdae61', 'ffffbf', 'a6d96a',       '1a9641']  
+}, 'landsat 8 image count (2020)');  
+  
+// Center the map at that point.  
+Map.centerObject(lisbonPoint, 5);

          Run the command and zoom out. If the count of images over the entire Earth is viewed, the resulting map should look like Fig. F4.1.2. The created map data may take a few minutes to fully load in.

          -

          -

          Fig. F4.1.2 The number of Landsat 8 images acquired during 2020

          +
          +
          +

          +

          Fig. F4.1.2 The number of Landsat 8 images acquired during 2020

          +
          +

          Note the checkered pattern, somewhat reminiscent of a Mondrian painting. To understand why the image looks this way, it is useful to consider the overlapping image footprints. As Landsat passes over, each image is wide enough to produce substantial “sidelap” with the images from the adjacent paths, which are collected at different dates according to the satellite’s orbit schedule. In the north-south direction, there is also some overlap to ensure that there are no gaps in the data. Because these are served as distinct images and stored distinctly in Earth Engine, you will find that there can be two images from the same day with the same value for points in these overlap areas. Depending on the purposes of a study, you might find a way to ignore the duplicate pixel values during the analysis process.

          -

          You might have noticed that we summarized a single band from the original ImageCollection to ensure that the resulting image would give a single count in each pixel. The count reducer operates on every band passed to it. Since every image has the same number of bands, passing an ImageCollection of all seven Landsat bands to the count reducer would have returned seven identical values of 16 for every point. To limit any confusion from seeing the same number seven times, we selected one of the bands from each image in the collection. In your own work, you might want to use a different reducer, such as a median operation, that would give different, useful answers for each band. A few of these reducers are described below.

          +

          You might have noticed that we summarized a single band from the original ImageCollection to ensure that the resulting image would give a single count in each pixel. The count reducer operates on every band passed to it. Since every image has the same number of bands, passing an ImageCollection of all seven Landsat bands to the count reducer would have returned seven identical values of 16 for every point. To limit any confusion from seeing the same number seven times, we selected one of the bands from each image in the collection. In your own work, you might want to use a different reducer, such as a median operation, that would give different, useful answers for each band. A few of these reducers are described below.

          @@ -653,22 +758,25 @@ Note
          -

          Code Checkpoint F41b. The book’s repository contains a script that shows what your code should look like at this point.

          +

          Code Checkpoint F41b. The book’s repository contains a script that shows what your code should look like at this point.

          -

          ## Reducing Image Collections to Understand Band Values

          -

          As we have seen, you could click at any point on Earth’s surface and see both the number of Landsat images recorded there in 2020 and the values of any image in any band through time. This is impressive and perhaps mind-bending, given the enormous amount of data in play. In this section and the next, we will explore two ways to summarize the numerical values of the bands—one straightforward way and one more complex but highly powerful way to see what information is contained in image collections.

          -

          First, we will make a new layer that represents the mean value of each band in every pixel across every image from 2020 for the filtered set, add this layer to the layer set, and explore again with the Inspector. The previous section’s count reducer was called directly using a sort of simple shorthand; that could be done similarly here by calling mean on the assembled bands. In this example, we will use the reducer to get the mean using the more general reduce call. Continue pasting the code below into the same script.

          -

          // Zoom to an informative scale for the code that follows.
          -Map.centerObject(lisbonPoint, 10);

          -

          // Add a mean composite image.
          -var meanFilteredIC = filteredIC.reduce(ee.Reducer.mean());
          -Map.addLayer(meanFilteredIC, {},   ‘Mean values within image collection’);

          -

          Now, let’s look at the median value for each band among all the values gathered in 2020. Using the code below, calculate the median and explore the image with the Inspector. Compare this image briefly to the mean image by eye and by clicking in a few pixels in the Inspector. They should have different values, but in most places they will look very similar.

          -

          // Add a median composite image.
          -var medianFilteredIC = filteredIC.reduce(ee.Reducer.median());
          -Map.addLayer(medianFilteredIC, {},   ‘Median values within image collection’);

          -

          There is a wide range of reducers available in Earth Engine. If you are curious about which reducers can be used to summarize band values across a collection of images, use the Docs tab in the Code Editor to list all reducers and look for those beginning with ee.Reducer.

          +
          +
          +

          7.3 Reducing Image Collections to Understand Band Values

          +

          As we have seen, you could click at any point on Earth’s surface and see both the number of Landsat images recorded there in 2020 and the values of any image in any band through time. This is impressive and perhaps mind-bending, given the enormous amount of data in play. In this section and the next, we will explore two ways to summarize the numerical values of the bands—one straightforward way and one more complex but highly powerful way to see what information is contained in image collections.

          +

          First, we will make a new layer that represents the mean value of each band in every pixel across every image from 2020 for the filtered set, add this layer to the layer set, and explore again with the Inspector. The previous section’s count reducer was called directly using a sort of simple shorthand; that could be done similarly here by calling mean on the assembled bands. In this example, we will use the reducer to get the mean using the more general reduce call. Continue pasting the code below into the same script.

          +
          // Zoom to an informative scale for the code that follows.  
+Map.centerObject(lisbonPoint, 10);  
+  
+// Add a mean composite image.  
+var meanFilteredIC = filteredIC.reduce(ee.Reducer.mean());  
+Map.addLayer(meanFilteredIC, {},   'Mean values within image collection');
          +

          Now, let’s look at the median value for each band among all the values gathered in 2020. Using the code below, calculate the median and explore the image with the Inspector. Compare this image briefly to the mean image by eye and by clicking in a few pixels in the Inspector. They should have different values, but in most places they will look very similar.

          +
          // Add a median composite image.  
+var medianFilteredIC = filteredIC.reduce(ee.Reducer.median());  
+Map.addLayer(medianFilteredIC, {},   'Median values within image collection');
          +

          There is a wide range of reducers available in Earth Engine. If you are curious about which reducers can be used to summarize band values across a collection of images, use the Docs tab in the Code Editor to list all reducers and look for those beginning with ee.Reducer.

          @@ -683,27 +791,36 @@ Note
          -
          -

          7.2 Compute Multiple Percentile Images for an Image Collection

          -

          One particularly useful reducer that can help you better understand the variability of values in image collections is ee.Reducer.percentile. The nth percentile gives the value that is the nth largest in a set. In this context, you can imagine accessing all of the values for a given band in a given ImageCollection for a given pixel and sorting them. The 30th percentile, for example, is the value 30% of the way along the list from smallest to largest. This provides an easy way to explore the variability of the values in image collections by computing a cumulative density function of values on a per-pixel basis. The following code shows how we can calculate a single 30th percentile on a per-pixel and per-band basis for our Landsat 8 ImageCollection. Continue pasting the code below into the same script.

          -

          // compute a single 30% percentile
          -var p30 = filteredIC.reduce(ee.Reducer.percentile([30]));

          -

          Map.addLayer(p30, {
          -   min: 0.05,
          -   max: 0.35}, ‘30%’);

          -

          -

          Fig. F4.1.3 Landsat 8 TOA reflectance 30th percentile image computed for ImageCollection with images acquired during 2020 

          +
          +

          7.4 Compute Multiple Percentile Images for an Image Collection

          +

          One particularly useful reducer that can help you better understand the variability of values in image collections is ee.Reducer.percentile. The nth percentile gives the value that is the nth largest in a set. In this context, you can imagine accessing all of the values for a given band in a given ImageCollection for a given pixel and sorting them. The 30th percentile, for example, is the value 30% of the way along the list from smallest to largest. This provides an easy way to explore the variability of the values in image collections by computing a cumulative density function of values on a per-pixel basis. The following code shows how we can calculate a single 30th percentile on a per-pixel and per-band basis for our Landsat 8 ImageCollection. Continue pasting the code below into the same script.

          +
          // compute a single 30% percentile  
+var p30 = filteredIC.reduce(ee.Reducer.percentile([30]));  
+  
+Map.addLayer(p30, {  
+   min: 0.05,  
+   max: 0.35}, '30%');
          +
          +
          +

          +

          Fig. F4.1.3 Landsat 8 TOA reflectance 30th percentile image computed for ImageCollection with images acquired during 2020

          +
          +

          We can see that the resulting composite image (Fig. 4.1.3) has almost no cloudy pixels present for this area. This happens because cloudy pixels usually have higher reflectance values. At the lowest end of the values, other unwanted effects like cloud or hill shadows typically have very low reflectance values. This is why this 30th percentile composite image looks so much cleaner than the mean composite image (meanFilteredIC) calculated earlier. Note that the reducers operate per pixel: adjacent pixels are drawn from different images. This means that one pixel’s value could be taken from an image from one date, and the adjacent pixel’s value drawn from an entirely different period. Although, like the mean and median images, percentile images such as that seen in Fig. F4.1.3 never existed on a single day, composite images allow us to view Earth’s surface without the noise that can make analysis difficult.

          -

          We can explore the range of values in an entire ImageCollection by viewing a series of increasingly bright percentile images, as shown in Fig. F4.1.4. Paste and run the following code.

          -

          var percentiles = [0, 10, 20, 30, 40, 50, 60, 70, 80];

          -

          // let’s compute percentile images and add them as separate layers
          -percentiles.map(function(p) {   var image = filteredIC.reduce(ee.Reducer.percentile([p]));   Map.addLayer(image, {
          -       min: 0.05,
          -       max: 0.35   }, p + ‘%’);
          -});

          -

          Note that the code adds every percentile image as a separate map layer, so you need to go to the Layers control and show/hide different layers to explore differences. Here, we can see that low-percentile composite images depict darker, low-reflectance land features, such as water and cloud or hill shadows, while higher-percentile composite images (>70% in our example) depict clouds and any other atmospheric or land effects corresponding to bright reflectance values.

          -

          -

          Fig. F4.1.4 Landsat 8 TOA reflectance percentile composite images

          +

          We can explore the range of values in an entire ImageCollection by viewing a series of increasingly bright percentile images, as shown in Fig. F4.1.4. Paste and run the following code.

          +

          var percentiles = [0, 10, 20, 30, 40, 50, 60, 70, 80];

          +
          // let's compute percentile images and add them as separate layers  
+percentiles.map(function(p) {   var image = filteredIC.reduce(ee.Reducer.percentile([p]));   Map.addLayer(image, {  
+       min: 0.05,  
+       max: 0.35   }, p + '%');  
+});
          +

          Note that the code adds every percentile image as a separate map layer, so you need to go to the Layers control and show/hide different layers to explore differences. Here, we can see that low-percentile composite images depict darker, low-reflectance land features, such as water and cloud or hill shadows, while higher-percentile composite images (>70% in our example) depict clouds and any other atmospheric or land effects corresponding to bright reflectance values.

          +
          +
          +

          +

          Fig. F4.1.4 Landsat 8 TOA reflectance percentile composite images

          +
          +

          Earth Engine provides a very rich API, allowing users to explore image collections to better understand the extent and variability of data in space, time, and across bands, as well as tools to analyze values stored in image collections in a frequency domain. Exploring these values in different forms should be the first step of any study before developing data analysis algorithms.

          @@ -715,7 +832,7 @@ Note
          -

          Code Checkpoint F41d. The book’s repository contains a script that shows what your code should look like at this point.

          +

          Code Checkpoint F41d. The book’s repository contains a script that shows what your code should look like at this point.

          @@ -723,8 +840,8 @@ Note

          Synthesis

          In the example above, the 30th percentile composite image would be useful for typical studies that need cloud-free data for analysis. The “best” composite to use, however, will depend on the goal of a study, the characteristics of the given data set, and the location being viewed. You can imagine choosing different percentile composite values if exploring image collections over the Sahara Desert or over Congo, where cloud frequency would vary substantially (Wilson et al. 2016).

          Assignment 1. Noting that your own interpretation of what constitutes a good composite is subjective, create a series of composites of a different location, or perhaps a pair of locations, for a given set of dates.

          -

          Assignment 2. Filter to create a relevant data set—for example, for Landsat 8 or Sentinel-2 over an agricultural growing season. Create percentile composites for a given location. Which image composite is the most satisfying, and what type of project do you have in mind when giving that response?

          -

          Assignment 3. Do you think it is possible to generalize about the relationship between the time window of an ImageCollection and the percentile value that will be the most useful for a given project, or will every region need to be inspected separately?

          +

          Assignment 2. Filter to create a relevant data set—for example, for Landsat 8 or Sentinel-2 over an agricultural growing season. Create percentile composites for a given location. Which image composite is the most satisfying, and what type of project do you have in mind when giving that response?

          +

          Assignment 3. Do you think it is possible to generalize about the relationship between the time window of an ImageCollection and the percentile value that will be the most useful for a given project, or will every region need to be inspected separately?

          Conclusion

          @@ -736,25 +853,33 @@ Note
          -

          8 Aggregating Images for Time Series

          -

          ::: {.callout-tip} # Chapter Information

          +

          8 Aggregating Images for Time Series

          +
          +
          +
          + +
          +
          +Chapter Information +
          +
          +

          Author

          -

          Ujaval Gandhi

          +

          Ujaval Gandhi

          Overview

          -

           

          -

          Many remote sensing datasets consist of repeated observations over time. The interval between observations can vary widely. The Global Precipitation Measurement dataset, for example, produces observations of rain and snow worldwide every three hours. The Climate Hazards Group InfraRed Precipitation with Station (CHIRPS) project produces a gridded global dataset at the daily level and also for each five-day period. The Landsat 8 mission produces a new scene of each location on Earth every 16 days. With its constellation of two satellites, the Sentinel-2 mission images every location every five days.

          -

          Many applications, however, require computing aggregations of data at time intervals different from those at which the datasets were produced. For example, for determining rainfall anomalies, it is useful to compare monthly rainfall against a long-period monthly average.

          -

          While individual scenes are informative, many days are cloudy, and it is useful to build a robust cloud-free time series for many applications. Producing less cloudy or even cloud-free composites can be done by aggregating data to form monthly, seasonal, or yearly composites built from individual scenes. For example, if you are interested in detecting long-term changes in an urban landscape, creating yearly median composites can enable you to detect change patterns across long time intervals with less worry about day-to-day noise.

          -

          This chapter will cover the techniques for aggregating individual images from a time  series at a chosen interval. We will take the CHIRPS time series of rainfall for one year and aggregate it to create a monthly rainfall time series.

          +

          Many remote sensing datasets consist of repeated observations over time. The interval between observations can vary widely. The Global Precipitation Measurement dataset, for example, produces observations of rain and snow worldwide every three hours. The Climate Hazards Group InfraRed Precipitation with Station (CHIRPS) project produces a gridded global dataset at the daily level and also for each five-day period. The Landsat 8 mission produces a new scene of each location on Earth every 16 days. With its constellation of two satellites, the Sentinel-2 mission images every location every five days.

          +

          Many applications, however, require computing aggregations of data at time intervals different from those at which the datasets were produced. For example, for determining rainfall anomalies, it is useful to compare monthly rainfall against a long-period monthly average.

          +

          While individual scenes are informative, many days are cloudy, and it is useful to build a robust cloud-free time series for many applications. Producing less cloudy or even cloud-free composites can be done by aggregating data to form monthly, seasonal, or yearly composites built from individual scenes. For example, if you are interested in detecting long-term changes in an urban landscape, creating yearly median composites can enable you to detect change patterns across long time intervals with less worry about day-to-day noise.

          +

          This chapter will cover the techniques for aggregating individual images from a time series at a chosen interval. We will take the CHIRPS time series of rainfall for one year and aggregate it to create a monthly rainfall time series.

          Learning Outcomes

            -
          • Using the Earth Engine API to work with dates.
            • -
          • Aggregating values from an ImageCollection to calculate monthly, seasonal, or yearly images.
            • +
          • Using the Earth Engine API to work with dates.
            • +
          • Aggregating values from an ImageCollection to calculate monthly, seasonal, or yearly images.
          • Plotting the aggregated time series at a given location.
          @@ -762,29 +887,35 @@ Note

          Assumes you know how to:

          • Import images and image collections, filter, and visualize (Part F1).
            • -
          • ​​Create a graph using ui.Chart (Chap. F1.3).
            • -
          • Write a function and map it over an ImageCollection (Chap. F4.0).
            • -
          • Summarize an ImageCollection with reducers (Chap. F4.0, Chap. F4.1).
            • +
          • ​​Create a graph using ui.Chart (Chap. F1.3).
            • +
          • Write a function and map it over an ImageCollection (Chap. F4.0).
            • +
          • Summarize an ImageCollection with reducers (Chap. F4.0, Chap. F4.1).
          • Inspect an Image and an ImageCollection, as well as their properties (Chap. F4.1).
          + +

          Introduction

          -

          CHIRPS is a high-resolution global gridded rainfall dataset that combines satellite-measured precipitation with ground station data in a consistent, long time-series dataset. The data are provided by the University of California, Santa Barbara, and are available from 1981 to the present. This dataset is extremely useful in drought monitoring and assessing global environmental change over land. The satellite data are calibrated with ground station observations to create the final product.

          -

          In this exercise, we will work with the CHIRPS dataset using the pentad. A pentad represents the grouping of five days. There are six pentads in a calendar month, with five pentads of exactly five days each and one pentad with the remaining three to six days of the month. Pentads reset at the beginning of each month, and the first day of every month is the start of a new pentad. Values at a given pixel in the CHIRPS dataset represent the total precipitation in millimeters over the pentad.

          +

          CHIRPS is a high-resolution global gridded rainfall dataset that combines satellite-measured precipitation with ground station data in a consistent, long time-series dataset. The data are provided by the University of California, Santa Barbara, and are available from 1981 to the present. This dataset is extremely useful in drought monitoring and assessing global environmental change over land. The satellite data are calibrated with ground station observations to create the final product.

          +

          In this exercise, we will work with the CHIRPS dataset using the pentad. A pentad represents the grouping of five days. There are six pentads in a calendar month, with five pentads of exactly five days each and one pentad with the remaining three to six days of the month. Pentads reset at the beginning of each month, and the first day of every month is the start of a new pentad. Values at a given pixel in the CHIRPS dataset represent the total precipitation in millimeters over the pentad.

          8.1 Filtering an Image Collection

          -

          We will start by accessing the CHIRPS Pentad collection and filtering it to create a time series for a single year.

          -

          var chirps = ee.ImageCollection(‘UCSB-CHG/CHIRPS/PENTAD’);
          -var startDate = ‘2019-01-01’;
          -var endDate = ‘2020-01-01’;
          -var yearFiltered = chirps.filter(ee.Filter.date(startDate, endDate));

          +

          We will start by accessing the CHIRPS Pentad collection and filtering it to create a time series for a single year.

          +

          var chirps = ee.ImageCollection(‘UCSB-CHG/CHIRPS/PENTAD’);
          +var startDate = ‘2019-01-01’;
          +var endDate = ‘2020-01-01’;
          +var yearFiltered = chirps.filter(ee.Filter.date(startDate, endDate));

          print(yearFiltered, ‘Date-filtered CHIRPS images’);

          -

          The CHIRPS collection contains one image for every pentad. The filtered collection above is filtered to contain one year, which equates to 72 global images. If you expand the printed collection in the Console, you will be able to see the metadata for individual images; note that their date stamps indicate that they are spaced evenly every five days (Fig. F4.2.1).

          -

          -

          Fig. F4.2.1 CHIRPS time series for one year

          -

          Each image’s pixel values store the total precipitation during the pentad. Without aggregation to a period that matches other datasets, these layers are not very useful. For hydrological analysis, we typically need the total precipitation for each month or for a season. Let’s aggregate this collection so that we have 12 images—one image per month, with pixel values that represent the total precipitation for that month.

          +

          The CHIRPS collection contains one image for every pentad. The filtered collection above is filtered to contain one year, which equates to 72 global images. If you expand the printed collection in the Console, you will be able to see the metadata for individual images; note that their date stamps indicate that they are spaced evenly every five days (Fig. F4.2.1).

          +
          +
          +

          +

          Fig. F4.2.1 CHIRPS time series for one year

          +
          +
          +

          Each image’s pixel values store the total precipitation during the pentad. Without aggregation to a period that matches other datasets, these layers are not very useful. For hydrological analysis, we typically need the total precipitation for each month or for a season. Let’s aggregate this collection so that we have 12 images—one image per month, with pixel values that represent the total precipitation for that month.

          @@ -795,29 +926,29 @@ Note
          -

          Code Checkpoint F42a. The book’s repository contains a script that shows what your code should look like at this point.

          +

          Code Checkpoint F42a. The book’s repository contains a script that shows what your code should look like at this point.

          8.2 Working with Dates

          -

          To aggregate the time series, we need to learn how to create and manipulate dates programmatically. This section covers some functions from the ee.Date module that will be useful.

          -

          The Earth Engine API has a function called ee.Date.fromYMD that is designed to create a date object from year, month, and day values. The following code snippet shows how to define a variable containing the year value and create a date object from it. Paste the following code in a new script:

          -

          var chirps = ee.ImageCollection(‘UCSB-CHG/CHIRPS/PENTAD’);
          -var year = 2019;
          -var startDate = ee.Date.fromYMD(year, 1, 1);

          -

          Now, let’s determine how to create an end date in order to be able to specify a desired time interval. The preferred way to create a date relative to another date is using the advance function. It takes two parameters—a delta value and the unit of time—and returns a new date. The code below shows how to create a date one year in the future from a given date. Paste it into your script.

          -

          var endDate = startDate.advance(1, ‘year’);

          -

          Next, paste the code below to perform filtering of the CHIRPS data using these calculated dates. After running it, check that you had accurately set the dates by looking for the dates of the images inside the printed result..

          -

          var yearFiltered = chirps
          -   .filter(ee.Filter.date(startDate, endDate));
          +

          To aggregate the time series, we need to learn how to create and manipulate dates programmatically. This section covers some functions from the ee.Date module that will be useful.

          +

          The Earth Engine API has a function called ee.Date.fromYMD that is designed to create a date object from year, month, and day values. The following code snippet shows how to define a variable containing the year value and create a date object from it. Paste the following code in a new script:

          +

          var chirps = ee.ImageCollection(‘UCSB-CHG/CHIRPS/PENTAD’);
          +var year = 2019;
          +var startDate = ee.Date.fromYMD(year, 1, 1);

          +

          Now, let’s determine how to create an end date in order to be able to specify a desired time interval. The preferred way to create a date relative to another date is using the advance function. It takes two parameters—a delta value and the unit of time—and returns a new date. The code below shows how to create a date one year in the future from a given date. Paste it into your script.

          +

          var endDate = startDate.advance(1, ‘year’);

          +

          Next, paste the code below to perform filtering of the CHIRPS data using these calculated dates. After running it, check that you had accurately set the dates by looking for the dates of the images inside the printed result..

          +

          var yearFiltered = chirps
          +.filter(ee.Filter.date(startDate, endDate));
          print(yearFiltered, ‘Date-filtered CHIRPS images’);

          -

          Another date function that is very commonly used across Earth Engine is millis. This function takes a date object and returns the number of milliseconds since the arbitrary reference date of the start of the year 1970: 1970-01-01T00:00:00Z. This is known as the “Unix Timestamp”; it is a standard way to convert dates to numbers and allows for easy comparison between dates with high precision. Earth Engine objects store the timestamps for images and features in special properties called system:time_start and system:time_end. Both of these properties need to be supplied with a number instead of dates, and the millis function can help you do that. You can print the result of calling this function and check for yourself.

          +

          Another date function that is very commonly used across Earth Engine is millis. This function takes a date object and returns the number of milliseconds since the arbitrary reference date of the start of the year 1970: 1970-01-01T00:00:00Z. This is known as the “Unix Timestamp”; it is a standard way to convert dates to numbers and allows for easy comparison between dates with high precision. Earth Engine objects store the timestamps for images and features in special properties called system:time_start and system:time_end. Both of these properties need to be supplied with a number instead of dates, and the millis function can help you do that. You can print the result of calling this function and check for yourself.

          print(startDate, ‘Start date’);
          print(endDate, ‘End date’);

          print(‘Start date as timestamp’, startDate.millis());
          print(‘End date as timestamp’, endDate.millis());

          -

          We will use the millis function in the next section when we need to set the system:time_start and system:time_end properties of the aggregated images.

          +

          We will use the millis function in the next section when we need to set the system:time_start and system:time_end properties of the aggregated images.

          @@ -828,33 +959,37 @@ Note
          -

          Code Checkpoint F42b. The book’s repository contains a script that shows what your code should look like at this point.

          +

          Code Checkpoint F42b. The book’s repository contains a script that shows what your code should look like at this point.

          8.3 Aggregating Images

          -

          Now we can start aggregating the pentads into monthly sums. The process of aggregation has two fundamental steps. The first is to determine the beginning and ending dates of one time interval (in this case, one month), and the second is to sum up all of the values (in this case, the pentads) that fall within each interval. To begin, we can envision that the resulting series will contain 12 images. To prepare to create an image for each month, we create an ee.List of values from 1 to 12. We can use the ee.List.sequence function, as first presented in Chap. F1.0, to create the list of items of type ee.Number. Continuing with the script of the previous section, paste the following code:

          -

          // Aggregate this time series to compute monthly images.
          -// Create a list of months
          -var months = ee.List.sequence(1, 12);

          -

          Next, we write a function that takes a single month as the input and returns an aggregated image for that month. Given beginningMonth as an input parameter, we first create a start and end date for that month based on the year and month variables. Then we filter the collection to find all images for that month. To create a monthly precipitation image, we apply ee.Reducer.sum to reduce the six pentad images for a month to a single image holding the summed value across the pentads. We also expressly set the timestamp properties system:time_start and system:time_end of the resulting summed image. We can also set year and month, which will help us filter the resulting collection later.

          -

          // Write a function that takes a month number
          -// and returns a monthly image.
          -var createMonthlyImage = function(beginningMonth) {   var startDate = ee.Date.fromYMD(year, beginningMonth, 1);   var endDate = startDate.advance(1, ‘month’);   var monthFiltered = yearFiltered
          -       .filter(ee.Filter.date(startDate, endDate));   // Calculate total precipitation.   var total = monthFiltered.reduce(ee.Reducer.sum());   return total.set({       ‘system:time_start’: startDate.millis(),       ‘system:time_end’: endDate.millis(),       ‘year’: year,       ‘month’: beginningMonth   });
          -};

          -

          We now have an ee.List containing items of type ee.Number from 1 to 12, with a function that can compute a monthly aggregated image for each month number. All that is left to do is to map the function over the list. As described in Chaps. F4.0 and F4.1, the map function passes over each image in the list and runs createMonthlyImage. The function first receives the number “1” and executes, returning an image to Earth Engine. Then it runs on the number “2”, and so on for all 12 numbers. The result is a list of monthly images for each month of the year.

          -

          // map() the function on the list of months
          -// This creates a list with images for each month in the list
          -var monthlyImages = months.map(createMonthlyImage);

          -

          We can create an ImageCollection from this ee.List of images using the ee.ImageCollection.fromImages function.

          -

          // Create an ee.ImageCollection.
          -var monthlyCollection = ee.ImageCollection.fromImages(monthlyImages);
          -print(monthlyCollection);

          -

          We have now successfully computed an aggregated collection from the source ImageCollection by filtering, mapping, and reducing, as described in Chaps. F4.0 and F4.1. Expand the printed collection in the Console and you can verify that we now have 12 images in the newly created ImageCollection (Fig. F4.2.2).

          -

          -

          Fig. F4.2.2 Aggregated time series

          +

          Now we can start aggregating the pentads into monthly sums. The process of aggregation has two fundamental steps. The first is to determine the beginning and ending dates of one time interval (in this case, one month), and the second is to sum up all of the values (in this case, the pentads) that fall within each interval. To begin, we can envision that the resulting series will contain 12 images. To prepare to create an image for each month, we create an ee.List of values from 1 to 12. We can use the ee.List.sequence function, as first presented in Chap. F1.0, to create the list of items of type ee.Number. Continuing with the script of the previous section, paste the following code:

          +
          // Aggregate this time series to compute monthly images.  
+// Create a list of months  
+var months = ee.List.sequence(1, 12);
          +

          Next, we write a function that takes a single month as the input and returns an aggregated image for that month. Given beginningMonth as an input parameter, we first create a start and end date for that month based on the year and month variables. Then we filter the collection to find all images for that month. To create a monthly precipitation image, we apply ee.Reducer.sum to reduce the six pentad images for a month to a single image holding the summed value across the pentads. We also expressly set the timestamp properties system:time_start and system:time_end of the resulting summed image. We can also set year and month, which will help us filter the resulting collection later.

          +
          // Write a function that takes a month number  
+// and returns a monthly image.  
+var createMonthlyImage = function(beginningMonth) {   var startDate = ee.Date.fromYMD(year, beginningMonth, 1);   var endDate = startDate.advance(1, 'month');   var monthFiltered = yearFiltered  
+       .filter(ee.Filter.date(startDate, endDate));   // Calculate total precipitation.   var total = monthFiltered.reduce(ee.Reducer.sum());   return total.set({       'system:time_start': startDate.millis(),       'system:time_end': endDate.millis(),       'year': year,       'month': beginningMonth   });  
+};
          +

          We now have an ee.List containing items of type ee.Number from 1 to 12, with a function that can compute a monthly aggregated image for each month number. All that is left to do is to map the function over the list. As described in Chaps. F4.0 and F4.1, the map function passes over each image in the list and runs createMonthlyImage. The function first receives the number “1” and executes, returning an image to Earth Engine. Then it runs on the number “2”, and so on for all 12 numbers. The result is a list of monthly images for each month of the year.

          +
          // map() the function on the list of months  
+// This creates a list with images for each month in the list  
+var monthlyImages = months.map(createMonthlyImage);
          +

          We can create an ImageCollection from this ee.List of images using the ee.ImageCollection.fromImages function.

          +
          // Create an ee.ImageCollection.  
+var monthlyCollection = ee.ImageCollection.fromImages(monthlyImages);  
+print(monthlyCollection);
          +

          We have now successfully computed an aggregated collection from the source ImageCollection by filtering, mapping, and reducing, as described in Chaps. F4.0 and F4.1. Expand the printed collection in the Console and you can verify that we now have 12 images in the newly created ImageCollection (Fig. F4.2.2).

          +
          +
          +

          +

          Fig. F4.2.2 Aggregated time series

          +
          +
          @@ -865,44 +1000,48 @@ Note
          -

          Code Checkpoint F42c. The book’s repository contains a script that shows what your code should look like at this point.

          +

          Code Checkpoint F42c. The book’s repository contains a script that shows what your code should look like at this point.

          8.4 Plotting Time Series

          One useful application of gridded precipitation datasets is to analyze rainfall patterns. We can plot a time-series chart for a location using the newly computed time series. We can plot the pixel value at any given point or polygon. Here we create a point geometry for a given coordinate. Continuing with the script of the previous section, paste the following code:

          -

          // Create a point with coordinates for the city of Bengaluru, India.
          -var point = ee.Geometry.Point(77.5946, 12.9716);

          -

          Earth Engine comes with a built-in ui.Chart.image.series function that can plot time series. In addition to the imageCollection and region parameters, we need to supply a scale value. The CHIRPS data catalog page indicates that the resolution of the data is ​​5566 meters, so we can use that as the scale. The resulting chart is printed in the Console.

          -

          var chart = ui.Chart.image.series({
          -   imageCollection: monthlyCollection,
          -   region: point,
          -   reducer: ee.Reducer.mean(),
          -   scale: 5566,
          +

          // Create a point with coordinates for the city of Bengaluru, India.  
+var point = ee.Geometry.Point(77.5946, 12.9716);
          +

          Earth Engine comes with a built-in ui.Chart.image.series function that can plot time series. In addition to the imageCollection and region parameters, we need to supply a scale value. The CHIRPS data catalog page indicates that the resolution of the data is ​​5566 meters, so we can use that as the scale. The resulting chart is printed in the Console.

          +

          var chart = ui.Chart.image.series({
          +imageCollection: monthlyCollection,
          +region: point,
          +reducer: ee.Reducer.mean(),
          +scale: 5566,
          });
          print(chart);

          -

          We can make the chart more informative by adding axis labels and a title. The setOptions function allows us to customize the chart using parameters from Google Charts. To customize the chart, paste the code below at the bottom of your script. The effect will be to see two charts in the editor: one with the old view of the data, and one with the customized chart.

          -

          var chart = ui.Chart.image.series({
          -   imageCollection: monthlyCollection,
          -   region: point,
          -   reducer: ee.Reducer.mean(),
          -   scale: 5566
          +

          We can make the chart more informative by adding axis labels and a title. The setOptions function allows us to customize the chart using parameters from Google Charts. To customize the chart, paste the code below at the bottom of your script. The effect will be to see two charts in the editor: one with the old view of the data, and one with the customized chart.

          +

          var chart = ui.Chart.image.series({
          +imageCollection: monthlyCollection,
          +region: point,
          +reducer: ee.Reducer.mean(),
          +scale: 5566
          }).setOptions({
          -   lineWidth: 1,
          -   pointSize: 3,
          -   title: ‘Monthly Rainfall at Bengaluru’,
          -   vAxis: {
          -       title: ‘Rainfall (mm)’   },
          -   hAxis: {
          -       title: ‘Month’,
          -       gridlines: {
          -           count: 12       }
          -   }
          +lineWidth: 1,
          +pointSize: 3,
          +title: ‘Monthly Rainfall at Bengaluru’,
          +vAxis: {
          +title: ‘Rainfall (mm)’ },
          +hAxis: {
          +title: ‘Month’,
          +gridlines: {
          +count: 12 }
          +}
          });print(chart);

          The customized chart (Fig. F4.2.3) shows the typical rainfall pattern in the city of Bengaluru, India. Bengaluru has a temperate climate, with pre-monsoon rains in April and May cooling down the city and a moderate monsoon season lasting from June to September.

          -

          -

          Fig. F4.2.3 Monthly rainfall chart

          +
          +
          +

          +

          Fig. F4.2.3 Monthly rainfall chart

          +
          +
          @@ -913,7 +1052,7 @@ Note
          -

          Code Checkpoint F42d. The book’s repository contains a script that shows what your code should look like at this point.

          +

          Code Checkpoint F42d. The book’s repository contains a script that shows what your code should look like at this point.

          @@ -921,20 +1060,22 @@ Note

          Synthesis

          Assignment 1. The CHIRPS collection contains data for 40 years. Aggregate the same collection to yearly images and create a chart for annual precipitation from 1981 to 2021 at your chosen location.

          Instead of creating a list of months and writing a function to create monthly images, we will create a list of years and write a function to create yearly images. The code snippet below will help you get started.

          -

          var chirps = ee.ImageCollection(‘UCSB-CHG/CHIRPS/PENTAD’);

          -

          // Create a list of years
          -var years = ee.List.sequence(1981, 2021);

          -

          // Write a function that takes a year number
          -// and returns a yearly image
          -var createYearlyImage = function(beginningYear) {   // Add your code
          -};

          -

          var yearlyImages = years.map(createYearlyImage);
          -var yearlyCollection = ee.ImageCollection.fromImages(yearlyImages);
          -print(yearlyCollection);

          +

          var chirps = ee.ImageCollection(‘UCSB-CHG/CHIRPS/PENTAD’);

          +
          // Create a list of years  
+var years = ee.List.sequence(1981, 2021);  
+  
+// Write a function that takes a year number  
+// and returns a yearly image  
+var createYearlyImage = function(beginningYear) {   // Add your code  
+};  
+  
+var yearlyImages = years.map(createYearlyImage);  
+var yearlyCollection = ee.ImageCollection.fromImages(yearlyImages);  
+print(yearlyCollection);

      • Conclusion

      -

      In this chapter, you learned how to aggregate a collection to months and plot the resulting time series for a location. This chapter also introduced useful functions for working with the dates that will be used across many different applications. You also learned how to iterate over a list using the map function. The technique of mapping a function over a list or collection is essential for processing data. Mastering this technique will allow you to scale your analysis using the parallel computing capabilities of Earth Engine.

      +

      In this chapter, you learned how to aggregate a collection to months and plot the resulting time series for a location. This chapter also introduced useful functions for working with the dates that will be used across many different applications. You also learned how to iterate over a list using the map function. The technique of mapping a function over a list or collection is essential for processing data. Mastering this technique will allow you to scale your analysis using the parallel computing capabilities of Earth Engine.

      References

      @@ -945,7 +1086,16 @@ print(yearlyCollection);

      9 Clouds and Image Compositing

      -

      ::: {.callout-tip} # Chapter Information

      +
      +
      +
      + +
      +
      +Chapter Information +
      +
      +

      Author

      :

      @@ -953,7 +1103,7 @@ print(yearlyCollection);

      Overview

      -

      The purpose of this chapter is to provide necessary context and demonstrate different approaches for image composite generation when using data quality flags, using an initial example of removing cloud cover. We will examine different filtering options, demonstrate an approach for cloud masking, and provide additional opportunities for image composite development. Pixel selection for composite development can exclude unwanted pixels—such as those impacted by cloud, shadow, and smoke or haze—and can also preferentially select pixels based upon proximity to a target date or a preferred sensor type.  

      +

      The purpose of this chapter is to provide necessary context and demonstrate different approaches for image composite generation when using data quality flags, using an initial example of removing cloud cover. We will examine different filtering options, demonstrate an approach for cloud masking, and provide additional opportunities for image composite development. Pixel selection for composite development can exclude unwanted pixels—such as those impacted by cloud, shadow, and smoke or haze—and can also preferentially select pixels based upon proximity to a target date or a preferred sensor type.

      Learning Outcomes

      @@ -971,95 +1121,131 @@ print(yearlyCollection);

    • Use band scaling factors (Chap. F3.1).
    • Perform pixel-based transformations (Chap. F3.1).
    • Use neighborhood-based image transformations (Chap. F3.2).
      • -
    • Write a function and map it over an ImageCollection (Chap. F4.0).
      • +
    • Write a function and map it over an ImageCollection (Chap. F4.0).
    • Summarize an ImageCollection with reducers (Chap. F4.0, Chap. F4.1).
    + +

    Introduction

    -

    In many respects, satellite remote sensing is an ideal source of data for monitoring large or remote regions. However, cloud cover is one of the most common limitations of optical sensors in providing continuous time series of data for surface mapping and monitoring. This is particularly relevant in tropical, polar, mountainous, and high-latitude areas, where clouds are often present. Many studies have addressed the extent to which cloudiness can restrict the monitoring of various regions (Zhu and Woodcock 2012, 2014; Eberhardt et al. 2016; Martins et al. 2018).

    -

    Clouds and cloud shadows reduce the view of optical sensors and completely block or obscure the spectral response from Earth’s surface (Cao et al. 2020). Working with pixels that are cloud-contaminated can significantly influence the accuracy and information content of products derived from a variety of remote sensing activities, including land cover classification, vegetation modeling, and especially change detection, where unscreened clouds might be mapped as false changes (Braaten et al. 2015, Zhu et al. 2015). Thus, the information provided by cloud detection algorithms is critical to exclude clouds and cloud shadows from subsequent processing steps.

    +

    In many respects, satellite remote sensing is an ideal source of data for monitoring large or remote regions. However, cloud cover is one of the most common limitations of optical sensors in providing continuous time series of data for surface mapping and monitoring. This is particularly relevant in tropical, polar, mountainous, and high-latitude areas, where clouds are often present. Many studies have addressed the extent to which cloudiness can restrict the monitoring of various regions (Zhu and Woodcock 2012, 2014; Eberhardt et al. 2016; Martins et al. 2018).

    +

    Clouds and cloud shadows reduce the view of optical sensors and completely block or obscure the spectral response from Earth’s surface (Cao et al. 2020). Working with pixels that are cloud-contaminated can significantly influence the accuracy and information content of products derived from a variety of remote sensing activities, including land cover classification, vegetation modeling, and especially change detection, where unscreened clouds might be mapped as false changes (Braaten et al. 2015, Zhu et al. 2015). Thus, the information provided by cloud detection algorithms is critical to exclude clouds and cloud shadows from subsequent processing steps.

    Historically, cloud detection algorithms derived the cloud information by considering a single date-image and sun illumination geometry (Irish et al. 2006, Huang et al. 2010). In contrast, current, more accurate cloud detection algorithms are based on the analysis of Landsat time series (Zhu and Woodcock 2014, Zhu and Helmer 2018). Cloud detection algorithms inform on the presence of clouds, cloud shadows, and other atmospheric conditions (e.g., presence of snow). The presence and extent of cloud contamination within a pixel is currently provided with Landsat and Sentinel-2 imagery as ancillary data via quality flags at the pixel level. Additionally, quality flags also inform on other acquisition-related conditions, including radiometric saturation and terrain occlusion, which enables us to assess the usefulness and convenience of inclusion of each pixel in subsequent analyses. The quality flags are ideally suited to reduce users’ manual supervision and maximize the automatic processing approaches.

    Most automated algorithms (for classification or change detection, for example) work best on images free of clouds and cloud shadows, that cover the full area without spatial or spectral inconsistencies. Thus, the image representation over the study area should be seamless, containing as few data gaps as possible. Image compositing techniques are primarily used to reduce the impact of clouds and cloud shadows, as well as aerosol contamination, view angle effects, and data volumes (White et al. 2014). Compositing approaches typically rely on the outputs of cloud detection algorithms and quality flags to include or exclude pixels from the resulting composite products (Roy et al. 2010). Epochal image composites help overcome the limited availability of cloud-free imagery in some areas, and are constructed by considering the pixels from all images acquired in a given period (e.g., season, year).

    -

    The information provided by the cloud masks and pixel flags guides the establishment of rules to rank the quality of the pixels based on the presence of and distance to clouds, cloud shadows, or atmospheric haze (Griffiths et al. 2010). Higher scores are assigned to pixels with more desirable conditions, based on the presence of clouds and also other acquisition circumstances, such as acquisition date or sensor. Those pixels with the highest scores are included in the subsequent composite development. Image compositing approaches enable users to define the rules that are most appropriate for their particular information needs and study area to generate imagery covering large areas instead of being limited to the analysis of single scenes (Hermosilla et al. 2015, Loveland and Dwyer 2012). Moreover, generating image composites at regular intervals (e.g., annually) allows for the analysis of long temporal series over large areas, fulfilling a critical information need for monitoring programs.

    +

    The information provided by the cloud masks and pixel flags guides the establishment of rules to rank the quality of the pixels based on the presence of and distance to clouds, cloud shadows, or atmospheric haze (Griffiths et al. 2010). Higher scores are assigned to pixels with more desirable conditions, based on the presence of clouds and also other acquisition circumstances, such as acquisition date or sensor. Those pixels with the highest scores are included in the subsequent composite development. Image compositing approaches enable users to define the rules that are most appropriate for their particular information needs and study area to generate imagery covering large areas instead of being limited to the analysis of single scenes (Hermosilla et al. 2015, Loveland and Dwyer 2012). Moreover, generating image composites at regular intervals (e.g., annually) allows for the analysis of long temporal series over large areas, fulfilling a critical information need for monitoring programs.

    The general workflow to generate a cloud-free composite involves:

      -
    1. Defining your area of interest (AOI).
      2. -
    2. Filtering (ee.Filter) the satellite ImageCollection to desired parameters.
      3. +
    3. Defining your area of interest (AOI).
      4. +
    4. Filtering (ee.Filter) the satellite ImageCollection to desired parameters.
    5. Applying a cloud mask.
    6. Reducing (ee.Reducer) the collection to generate a composite.
      7. -
    7. Using the GEE-BAP application to generate annual best-available-pixel image composites by globally combining multiple Landsat sensors and images.
      8. +
    8. Using the GEE-BAP application to generate annual best-available-pixel image composites by globally combining multiple Landsat sensors and images.

    Additional steps may be necessary to improve the composite generated. These steps will be explained in the following sections.

    -

    ## Cloud Filter and Cloud Mask

    -

    The first step is to define your AOI and center the map. The goal is to create a nationwide composite for the country of Colombia. We will use the Large Scale International Boundary (2017) simplified dataset from the US Department of State (USDOS), which contains polygons for all countries of the world.

    -

    // ———- Section 1 —————–

    -

    // Define the AOI.
    -var country = ee.FeatureCollection(‘USDOS/LSIB_SIMPLE/2017’)
    -   .filter(ee.Filter.equals(‘country_na’, ‘Colombia’));

    -

    // Center the Map. The second parameter is zoom level.
    -Map.centerObject(country, 5);

    -

    We will start creating a composite from the Landsat 8 collection. First, we define two time variables: startDate and endDate. Here, we will create a composite for the year 2019. Then, we will define a collection for the Landsat 8 Level 2, Collection 2, Tier 1 variable and filter it to our AOI and time period. We define and use a function to apply scaling factors to the Landsat 8 Collection 2 data.

    -

    // Define time variables.
    -var startDate = ‘2019-01-01’;
    -var endDate = ‘2019-12-31’;

    -

    // Load and filter the Landsat 8 collection.
    -var landsat8 = ee.ImageCollection(‘LANDSAT/LC08/C02/T1_L2’)
    -   .filterBounds(country)
    -   .filterDate(startDate, endDate);

    -

    // Apply scaling factors.
    -function applyScaleFactors(image) {   var opticalBands = image.select(‘SR_B.’).multiply(0.0000275).add(-       0.2);   var thermalBands = image.select(’ST_B.*’).multiply(0.00341802)
    -       .add(149.0);   return image.addBands(opticalBands, null, true)
    -       .addBands(thermalBands, null, true);
    -}

    -

    landsat8 = landsat8.map(applyScaleFactors);

    -

    Now, we can create a composite. We will use the median function, which has the same effect as writing reduce(ee.Reducer.median()) as seen in Chap. F4.0, to reduce our ImageCollection to a median composite. Add the resulting composite to the map using visualization parameters.

    -

    // Create composite.
    -var composite = landsat8.median().clip(country);

    -

    var visParams = {
    -   bands: [‘SR_B4’, ‘SR_B3’, ‘SR_B2’],
    -   min: 0,
    -   max: 0.2
    -};
    -Map.addLayer(composite, visParams, ‘L8 Composite’);

    -

    -

    Fig. F4.3.1 Landsat 8 surface reflectance 2019 median composite of Colombia

    -

    The resulting composite (Fig. F4.3.1) has lots of clouds, especially in the western, mountainous regions of Colombia. In tropical regions, it is very challenging to generate a high-quality, cloud-free composite without first filtering images for cloud cover, even if our collection is constrained to only include images acquired during the dry season. Therefore, let’s filter our collection by the CLOUD_COVER parameter to avoid cloudy images. We will start with images that have less than 50% cloud cover.

    -

    // Filter by the CLOUD_COVER property.
    -var landsat8FiltClouds = landsat8
    -   .filterBounds(country)
    -   .filterDate(startDate, endDate)
    -   .filter(ee.Filter.lessThan(‘CLOUD_COVER’, 50));

    -

    // Create a composite from the filtered imagery.
    -var compositeFiltClouds = landsat8FiltClouds.median().clip(country);

    -

    Map.addLayer(compositeFiltClouds, visParams,   ‘L8 Composite cloud filter’);

    -

    // Print size of collections, for comparison.
    -print(‘Size landsat8 collection’, landsat8.size());
    -print(‘Size landsat8FiltClouds collection’, landsat8FiltClouds.size());

    -

    -

    Fig. F4.3.2 Landsat 8 surface reflectance 2019 median composite of Colombia filtered by cloud cover less than 50%

    -

    This new composite (Fig. F4.3.2) looks slightly better than the previous one, but still very cloudy. Remember to turn off the first layer or adjust the transparency to visualize only this new composite. The code prints the size of these collections, using the size function) to see how many images were left out after we applied the cloud cover threshold. (There are 1201 images in the landsat8 collection, compared to 493 in the landsat8FiltClouds collection—a lot of scenes with cloud cover greater than or equal to 50%.)

    -

    Try adjusting the CLOUD_COVER threshold in the landsat8FiltClouds variable to different percentages and checking the results. For example, with 20% set as the threshold (Fig. F4.3.3), you can see that many parts of the country have image gaps. (Remember to turn off the first layer or adjust its transparency; you can also set the shown parameter in the Map.addLayer function to false so the layer does not automatically load). So there is a trade-off between a stricter cloud cover threshold and data availability. Additionally, even with a cloud filter, some tiles still present a large area cover of clouds.

    -

    -

    Fig. F4.3.3 Landsat 8 surface reflectance 2019 median composite of Colombia filtered by cloud cover less than 20%

    -

    This is due to persistent cloud cover in some regions of Colombia. However, a cloud mask can be applied to improve the results. The Landsat 8 Collection 2 contains a quality assessment (QA) band called QA_PIXEL that provides useful information on certain conditions within the data, and allows users to apply per-pixel filters. Each pixel in the QA band contains unsigned integers that represent bit-packed combinations of surface, atmospheric, and sensor conditions.

    -

    We will also make use of the QA_RADSAT band, which indicates which bands are radiometrically saturated. A pixel value of 1 means saturated, so we will be masking these pixels.

    -

    As described in Chap. F4.0, we will create a function to apply a cloud mask to an image, and then map this function over our collection. The mask is applied by using the updateMask function. This function “eliminates” undesired pixels from the analysis, i.e., makes them transparent, by taking the mask as the input. You will see that this cloud mask function (or similar versions) is used in other chapters of the book. Note: Remember to set the cloud cover threshold back to 50 in the landsat8FiltClouds variable.

    -

    // Define the cloud mask function.
    -function maskSrClouds(image) {   // Bit 0 - Fill   // Bit 1 - Dilated Cloud   // Bit 2 - Cirrus   // Bit 3 - Cloud   // Bit 4 - Cloud Shadow   var qaMask = image.select(‘QA_PIXEL’).bitwiseAnd(parseInt(‘11111’,       2)).eq(0);   var saturationMask = image.select(‘QA_RADSAT’).eq(0);   return image.updateMask(qaMask)
    -       .updateMask(saturationMask);
    -}

    -

    // Apply the cloud mask to the collection.
    -var landsat8FiltMasked = landsat8FiltClouds.map(maskSrClouds);

    -

    // Create a composite.
    -var landsat8compositeMasked = landsat8FiltMasked.median().clip(country);

    -

    Map.addLayer(landsat8compositeMasked, visParams, ‘L8 composite masked’);

    -

    -

    Fig. F4.3.4 Landsat 8 surface reflectance 2019 median composite of Colombia filtered by cloud cover less than 50% and with cloud mask applied

    -

    Because we are dealing with bits, in the maskSrClouds function we utilized the bitwiseAnd and parseInt functions. These are functions that serve the purpose of unpacking the bit information. A bitwise AND is a binary operation that takes two equal-length binary representations and performs the logical AND operation on each pair of corresponding bits. Thus, if both bits in the compared positions have the value 1, the bit in the resulting binary representation is 1 (1 × 1 = 1); otherwise, the result is 0 (1 × 0 = 0 and 0 × 0 = 0). The parseInt function parses a string argument (in our case, five-character string ‘11111’) and returns an integer of the specified numbering system, base 2.

    -

    The resulting composite (Fig. F4.3.4) shows masked clouds, and is more spatially exhaustive in coverage compared to previous composites (don’t forget to uncheck the previous layers). This is because, when compositing all the images into one, we are not taking cloudy pixels into account anymore; therefore, the resulting pixel is not cloud covered but an actual representation of the landscape. However, data gaps are still an issue due to cloud cover. If you do not specifically need an annual composite, a first approach is to create a two-year composite to try to mitigate the missing data issue, or to have a series of rules that allows for selecting pixels for that particular year (as in Sect. 3 below). Change the startDate variable to 2018-01-01 to include all images from 2018 and 2019 in the collection. How does the cloud-masked composite (Fig. F4.3.5) compare to the 2019 one?

    -

    -

    Fig. F4.3.5 One-year, startDate variable set to 2019-01-01, (left) and two-year, startDate variable set to 2018-01-01, (right) median composites with 50% cloud cover threshold and cloud mask applied

    +
    +
    +

    9.1 Cloud Filter and Cloud Mask

    +

    The first step is to define your AOI and center the map. The goal is to create a nationwide composite for the country of Colombia. We will use the Large Scale International Boundary (2017) simplified dataset from the US Department of State (USDOS), which contains polygons for all countries of the world.

    +
    // ---------- Section 1 -----------------  
+  
+// Define the AOI.  
+var country = ee.FeatureCollection('USDOS/LSIB_SIMPLE/2017')  
+   .filter(ee.Filter.equals('country_na', 'Colombia'));  
+  
+// Center the Map. The second parameter is zoom level.  
+Map.centerObject(country, 5);
    +

    We will start creating a composite from the Landsat 8 collection. First, we define two time variables: startDate and endDate. Here, we will create a composite for the year 2019. Then, we will define a collection for the Landsat 8 Level 2, Collection 2, Tier 1 variable and filter it to our AOI and time period. We define and use a function to apply scaling factors to the Landsat 8 Collection 2 data.

    +
    // Define time variables.  
+var startDate = '2019-01-01';  
+var endDate = '2019-12-31';  
+  
+// Load and filter the Landsat 8 collection.  
+var landsat8 = ee.ImageCollection('LANDSAT/LC08/C02/T1_L2')  
+   .filterBounds(country)  
+   .filterDate(startDate, endDate);  
+  
+// Apply scaling factors.  
+function applyScaleFactors(image) {   var opticalBands = image.select('SR_B.').multiply(0.0000275).add(-       0.2);   var thermalBands = image.select('ST_B.*').multiply(0.00341802)  
+       .add(149.0);   return image.addBands(opticalBands, null, true)  
+       .addBands(thermalBands, null, true);  
+}  
+  
+landsat8 = landsat8.map(applyScaleFactors);
    +

    Now, we can create a composite. We will use the median function, which has the same effect as writing reduce(ee.Reducer.median()) as seen in Chap. F4.0, to reduce our ImageCollection to a median composite. Add the resulting composite to the map using visualization parameters.

    +
    // Create composite.  
+var composite = landsat8.median().clip(country);  
+  
+var visParams = {  
+   bands: ['SR_B4', 'SR_B3', 'SR_B2'],  
+   min: 0,  
+   max: 0.2  
+};  
+Map.addLayer(composite, visParams, 'L8 Composite');
    +
    +
    +

    +

    Fig. F4.3.1 Landsat 8 surface reflectance 2019 median composite of Colombia

    +
    +
    +

    The resulting composite (Fig. F4.3.1) has lots of clouds, especially in the western, mountainous regions of Colombia. In tropical regions, it is very challenging to generate a high-quality, cloud-free composite without first filtering images for cloud cover, even if our collection is constrained to only include images acquired during the dry season. Therefore, let’s filter our collection by the CLOUD_COVER parameter to avoid cloudy images. We will start with images that have less than 50% cloud cover.

    +
    // Filter by the CLOUD_COVER property.  
+var landsat8FiltClouds = landsat8  
+   .filterBounds(country)  
+   .filterDate(startDate, endDate)  
+   .filter(ee.Filter.lessThan('CLOUD_COVER', 50));  
+  
+// Create a composite from the filtered imagery.  
+var compositeFiltClouds = landsat8FiltClouds.median().clip(country);  
+  
+Map.addLayer(compositeFiltClouds, visParams,   'L8 Composite cloud filter');  
+  
+// Print size of collections, for comparison.  
+print('Size landsat8 collection', landsat8.size());  
+print('Size landsat8FiltClouds collection', landsat8FiltClouds.size());
    +
    +
    +

    +

    Fig. F4.3.2 Landsat 8 surface reflectance 2019 median composite of Colombia filtered by cloud cover less than 50%

    +
    +
    +

    This new composite (Fig. F4.3.2) looks slightly better than the previous one, but still very cloudy. Remember to turn off the first layer or adjust the transparency to visualize only this new composite. The code prints the size of these collections, using the size function) to see how many images were left out after we applied the cloud cover threshold. (There are 1201 images in the landsat8 collection, compared to 493 in the landsat8FiltClouds collection—a lot of scenes with cloud cover greater than or equal to 50%.)

    +

    Try adjusting the CLOUD_COVER threshold in the landsat8FiltClouds variable to different percentages and checking the results. For example, with 20% set as the threshold (Fig. F4.3.3), you can see that many parts of the country have image gaps. (Remember to turn off the first layer or adjust its transparency; you can also set the shown parameter in the Map.addLayer function to false so the layer does not automatically load). So there is a trade-off between a stricter cloud cover threshold and data availability. Additionally, even with a cloud filter, some tiles still present a large area cover of clouds.

    +
    +
    +

    +

    Fig. F4.3.3 Landsat 8 surface reflectance 2019 median composite of Colombia filtered by cloud cover less than 20%

    +
    +
    +

    This is due to persistent cloud cover in some regions of Colombia. However, a cloud mask can be applied to improve the results. The Landsat 8 Collection 2 contains a quality assessment (QA) band called QA_PIXEL that provides useful information on certain conditions within the data, and allows users to apply per-pixel filters. Each pixel in the QA band contains unsigned integers that represent bit-packed combinations of surface, atmospheric, and sensor conditions.

    +

    We will also make use of the QA_RADSAT band, which indicates which bands are radiometrically saturated. A pixel value of 1 means saturated, so we will be masking these pixels.

    +

    As described in Chap. F4.0, we will create a function to apply a cloud mask to an image, and then map this function over our collection. The mask is applied by using the updateMask function. This function “eliminates” undesired pixels from the analysis, i.e., makes them transparent, by taking the mask as the input. You will see that this cloud mask function (or similar versions) is used in other chapters of the book. Note: Remember to set the cloud cover threshold back to 50 in the landsat8FiltClouds variable.

    +
    // Define the cloud mask function.  
+function maskSrClouds(image) {   // Bit 0 - Fill   // Bit 1 - Dilated Cloud   // Bit 2 - Cirrus   // Bit 3 - Cloud   // Bit 4 - Cloud Shadow   var qaMask = image.select('QA_PIXEL').bitwiseAnd(parseInt('11111',       2)).eq(0);   var saturationMask = image.select('QA_RADSAT').eq(0);   return image.updateMask(qaMask)  
+       .updateMask(saturationMask);  
+}  
+  
+// Apply the cloud mask to the collection.  
+var landsat8FiltMasked = landsat8FiltClouds.map(maskSrClouds);  
+  
+// Create a composite.  
+var landsat8compositeMasked = landsat8FiltMasked.median().clip(country);  
+  
+Map.addLayer(landsat8compositeMasked, visParams, 'L8 composite masked');
    +
    +
    +

    +

    Fig. F4.3.4 Landsat 8 surface reflectance 2019 median composite of Colombia filtered by cloud cover less than 50% and with cloud mask applied

    +
    +
    +

    Because we are dealing with bits, in the maskSrClouds function we utilized the bitwiseAnd and parseInt functions. These are functions that serve the purpose of unpacking the bit information. A bitwise AND is a binary operation that takes two equal-length binary representations and performs the logical AND operation on each pair of corresponding bits. Thus, if both bits in the compared positions have the value 1, the bit in the resulting binary representation is 1 (1 × 1 = 1); otherwise, the result is 0 (1 × 0 = 0 and 0 × 0 = 0). The parseInt function parses a string argument (in our case, five-character string ‘11111’) and returns an integer of the specified numbering system, base 2.

    +

    The resulting composite (Fig. F4.3.4) shows masked clouds, and is more spatially exhaustive in coverage compared to previous composites (don’t forget to uncheck the previous layers). This is because, when compositing all the images into one, we are not taking cloudy pixels into account anymore; therefore, the resulting pixel is not cloud covered but an actual representation of the landscape. However, data gaps are still an issue due to cloud cover. If you do not specifically need an annual composite, a first approach is to create a two-year composite to try to mitigate the missing data issue, or to have a series of rules that allows for selecting pixels for that particular year (as in Sect. 3 below). Change the startDate variable to 2018-01-01 to include all images from 2018 and 2019 in the collection. How does the cloud-masked composite (Fig. F4.3.5) compare to the 2019 one?

    +
    +
    +

    +

    Fig. F4.3.5 One-year, startDate variable set to 2019-01-01, (left) and two-year, startDate variable set to 2018-01-01, (right) median composites with 50% cloud cover threshold and cloud mask applied

    +
    +

    The resulting image has substantially fewer data gaps (you can zoom in to better see them). Again, if the time period is not a constraint for the creation of your composite, you can incorporate more images from a third year, and so on.

    @@ -1071,70 +1257,92 @@ Note
    -

    Code Checkpoint F43a. The book’s repository contains a script that shows what your code should look like at this point.

    +

    Code Checkpoint F43a. The book’s repository contains a script that shows what your code should look like at this point.

    -
    -

    9.1 Incorporating Data from Other Satellites

    -

    Another option to reduce the presence of data gaps in cloudy situations is to bring in imagery from other sensors acquired during the time period of interest. The Landsat collection spans multiple missions, which have continuously acquired uninterrupted data since 1972 at different acquisition dates. Next, we will try incorporating Landsat 7 Level 2, Collection 2, Tier 1 images from 2019 to fill the gaps in the 2019 Landsat 8 composite.

    -

    To generate a Landsat 7 composite, we apply similar steps to the ones we did for Landsat 8, so keep adding code to the same script from Sect. 1. First, define your Landsat 7 collection variable and the scaling function. Then, filter the collection, apply the cloud mask (since we know Colombia has persistent cloud cover), and apply the scaling function. Note that we will use the same cloud mask function defined above, since the bits information for Landsat 7 is the same as for Landsat 8. Finally, create the median composite. After pasting in the code below but before executing it, change the startDate variable back to 2019-01-01 in order to create a one-year composite of 2019.

    -

    // ———- Section 2 —————–

    -

    // Define Landsat 7 Level 2, Collection 2, Tier 1 collection.
    -var landsat7 = ee.ImageCollection(‘LANDSAT/LE07/C02/T1_L2’);

    -

    // Scaling factors for L7.
    -function applyScaleFactorsL7(image) {   var opticalBands = image.select(‘SR_B.’).multiply(0.0000275).add(-       0.2);   var thermalBand = image.select(‘ST_B6’).multiply(0.00341802).add(       149.0);   return image.addBands(opticalBands, null, true)
    -       .addBands(thermalBand, null, true);
    -}

    -

    // Filter collection, apply cloud mask, and scaling factors.
    -var landsat7FiltMasked = landsat7
    -   .filterBounds(country)
    -   .filterDate(startDate, endDate)
    -   .filter(ee.Filter.lessThan(‘CLOUD_COVER’, 50))
    -   .map(maskSrClouds)
    -   .map(applyScaleFactorsL7);

    -

    // Create composite.
    -var landsat7compositeMasked = landsat7FiltMasked
    -   .median()
    -   .clip(country);

    -

    Map.addLayer(landsat7compositeMasked,
    -   {
    -       bands: [‘SR_B3’, ‘SR_B2’, ‘SR_B1’],
    -       min: 0,
    -       max: 0.2   },   ‘L7 composite masked’);

    -

    -

    Fig. F4.3.6 One-year Landsat 7 median composite with 50% cloud cover threshold and cloud mask applied

    -

    Note that we used bands: [‘SR_B3’, ‘SR_B2’, ‘SR_B1’] to visualize the composite because Landsat 7 has different band designations. The sensors aboard each of the Landsat satellites were designed to acquire data in different ranges of frequencies along the electromagnetic spectrum. Whereas for Landsat 8, the red, green, and blue bands are B4, B3, and B2, respectively, for Landsat 7, these same bands are B3, B2, and B1, respectively.

    -

    You should see an image with systematic gaps like the one shown in Fig. F4.3.6 (remember to turn off the other layers, and zoom in to better see the data gaps). Landsat 7 was launched in 1999, but since 2003, the sensor has acquired and delivered data with data gaps caused by a scan line corrector (SLC) failure. Without an operating SLC, the sensor’s line of sight traces a zig-zag pattern along the satellite ground track, and, as a result, the imaged area is duplicated and some areas are missed. When the Level 1 data are processed, the duplicated areas are removed, leaving data gaps (Fig. F4.3.7). For more information about Landsat 7 and SLC error, please refer to the USGS Landsat 7 page. However, even with the SLC error, we can still use the Landsat 7 data in our composite. Now, let’s combine the Landsat 7 and 8 collections.

    -

    -

    Fig. F4.3.7 Landsat 7’s SLC-off condition. Source: USGS

    -

    Since Landsat 7 and 8 have different band designations, first we create a function to rename the bands from Landsat 7 to match the names used for Landsat 8 and map that function over our Landsat 7 collection.

    -

    // Since Landsat 7 and 8 have different band designations,
    -// let’s create a function to rename L7 bands to match to L8.
    -function rename(image) {   return image.select(
    -       [‘SR_B1’, ‘SR_B2’, ‘SR_B3’, ‘SR_B4’, ‘SR_B5’, ‘SR_B7’],
    -       [‘SR_B2’, ‘SR_B3’, ‘SR_B4’, ‘SR_B5’, ‘SR_B6’, ‘SR_B7’]);
    -}

    -

    // Apply the rename function.
    -var landsat7FiltMaskedRenamed = landsat7FiltMasked.map(rename);

    -

    If you print the first images of both the landsat7FiltMasked and landsat7FiltMaskedRenamed collections (Fig. F4.3.8), you will see that the bands got renamed, and not all bands got copied over (SR_ATMOS_OPACITY, SR_CLOUD_QA, SR_B6, etc.). To copy these additional bands, simply add them to the rename function. You will need to rename SR_B6 so it does not have the same name as the new band 5.

    -

    -

    Fig. F4.3.8 First images of landsat7FiltMasked and landsat7FiltMaskedRenamed, respectively

    -

    Now we merge the two collections using the merge function for ImageCollection and mapping over a function to cast the Landsat 7 input values to a 32-bit float using the toFloat function for consistency. To merge collections, the number and names of the bands must be the same in each collection. We use the select function (Chap. F1.1) to select the Landsat 8 bands to be the same as Landsat 7’s. When creating the new Landsat 7 and 8 composite, if we did not select these 6 bands, we would get an error message for trying to composite a collection that has 6 bands (Landsat 7) with a collection that has 19 bands (Landsat 8).

    -

    // Merge Landsat collections.
    -var landsat78 = landsat7FiltMaskedRenamed
    -   .merge(landsat8FiltMasked.select(
    -       [‘SR_B2’, ‘SR_B3’, ‘SR_B4’, ‘SR_B5’, ‘SR_B6’, ‘SR_B7’]))
    -   .map(function(img) {       return img.toFloat();
    -   });
    -print(‘Merged collections’, landsat78);

    -

    Now we have a collection with about 1000 images. Next, we will take the median of the values across the ImageCollection. 

    -

    // Create Landsat 7 and 8 image composite and add to the Map.
    -var landsat78composite = landsat78.median().clip(country);
    -Map.addLayer(landsat78composite, visParams, ‘L7 and L8 composite’);

    +
    +

    9.2 Incorporating Data from Other Satellites

    +

    Another option to reduce the presence of data gaps in cloudy situations is to bring in imagery from other sensors acquired during the time period of interest. The Landsat collection spans multiple missions, which have continuously acquired uninterrupted data since 1972 at different acquisition dates. Next, we will try incorporating Landsat 7 Level 2, Collection 2, Tier 1 images from 2019 to fill the gaps in the 2019 Landsat 8 composite.

    +

    To generate a Landsat 7 composite, we apply similar steps to the ones we did for Landsat 8, so keep adding code to the same script from Sect. 1. First, define your Landsat 7 collection variable and the scaling function. Then, filter the collection, apply the cloud mask (since we know Colombia has persistent cloud cover), and apply the scaling function. Note that we will use the same cloud mask function defined above, since the bits information for Landsat 7 is the same as for Landsat 8. Finally, create the median composite. After pasting in the code below but before executing it, change the startDate variable back to 2019-01-01 in order to create a one-year composite of 2019.

    +
    // ---------- Section 2 -----------------  
+  
+// Define Landsat 7 Level 2, Collection 2, Tier 1 collection.  
+var landsat7 = ee.ImageCollection('LANDSAT/LE07/C02/T1_L2');  
+  
+// Scaling factors for L7.  
+function applyScaleFactorsL7(image) {   var opticalBands = image.select('SR_B.').multiply(0.0000275).add(-       0.2);   var thermalBand = image.select('ST_B6').multiply(0.00341802).add(       149.0);   return image.addBands(opticalBands, null, true)  
+       .addBands(thermalBand, null, true);  
+}  
+  
+// Filter collection, apply cloud mask, and scaling factors.  
+var landsat7FiltMasked = landsat7  
+   .filterBounds(country)  
+   .filterDate(startDate, endDate)  
+   .filter(ee.Filter.lessThan('CLOUD_COVER', 50))  
+   .map(maskSrClouds)  
+   .map(applyScaleFactorsL7);  
+  
+// Create composite.  
+var landsat7compositeMasked = landsat7FiltMasked  
+   .median()  
+   .clip(country);  
+  
+Map.addLayer(landsat7compositeMasked,  
+   {  
+       bands: ['SR_B3', 'SR_B2', 'SR_B1'],  
+       min: 0,  
+       max: 0.2   },   'L7 composite masked');
    +
    +
    +

    +

    Fig. F4.3.6 One-year Landsat 7 median composite with 50% cloud cover threshold and cloud mask applied

    +
    +
    +

    Note that we used bands: [‘SR_B3’, ‘SR_B2’, ‘SR_B1’] to visualize the composite because Landsat 7 has different band designations. The sensors aboard each of the Landsat satellites were designed to acquire data in different ranges of frequencies along the electromagnetic spectrum. Whereas for Landsat 8, the red, green, and blue bands are B4, B3, and B2, respectively, for Landsat 7, these same bands are B3, B2, and B1, respectively.

    +

    You should see an image with systematic gaps like the one shown in Fig. F4.3.6 (remember to turn off the other layers, and zoom in to better see the data gaps). Landsat 7 was launched in 1999, but since 2003, the sensor has acquired and delivered data with data gaps caused by a scan line corrector (SLC) failure. Without an operating SLC, the sensor’s line of sight traces a zig-zag pattern along the satellite ground track, and, as a result, the imaged area is duplicated and some areas are missed. When the Level 1 data are processed, the duplicated areas are removed, leaving data gaps (Fig. F4.3.7). For more information about Landsat 7 and SLC error, please refer to the USGS Landsat 7 page. However, even with the SLC error, we can still use the Landsat 7 data in our composite. Now, let’s combine the Landsat 7 and 8 collections.

    +
    +
    +

    +

    Fig. F4.3.7 Landsat 7’s SLC-off condition. Source: USGS

    +
    +
    +

    Since Landsat 7 and 8 have different band designations, first we create a function to rename the bands from Landsat 7 to match the names used for Landsat 8 and map that function over our Landsat 7 collection.

    +
    // Since Landsat 7 and 8 have different band designations,  
+// let's create a function to rename L7 bands to match to L8.  
+function rename(image) {   return image.select(  
+       ['SR_B1', 'SR_B2', 'SR_B3', 'SR_B4', 'SR_B5', 'SR_B7'],  
+       ['SR_B2', 'SR_B3', 'SR_B4', 'SR_B5', 'SR_B6', 'SR_B7']);  
+}  
+  
+// Apply the rename function.  
+var landsat7FiltMaskedRenamed = landsat7FiltMasked.map(rename);
    +

    If you print the first images of both the landsat7FiltMasked and landsat7FiltMaskedRenamed collections (Fig. F4.3.8), you will see that the bands got renamed, and not all bands got copied over (SR_ATMOS_OPACITY, SR_CLOUD_QA, SR_B6, etc.). To copy these additional bands, simply add them to the rename function. You will need to rename SR_B6 so it does not have the same name as the new band 5.

    +
    +
    +

    +

    Fig. F4.3.8 First images of landsat7FiltMasked and landsat7FiltMaskedRenamed, respectively

    +
    +
    +

    Now we merge the two collections using the merge function for ImageCollection and mapping over a function to cast the Landsat 7 input values to a 32-bit float using the toFloat function for consistency. To merge collections, the number and names of the bands must be the same in each collection. We use the select function (Chap. F1.1) to select the Landsat 8 bands to be the same as Landsat 7’s. When creating the new Landsat 7 and 8 composite, if we did not select these 6 bands, we would get an error message for trying to composite a collection that has 6 bands (Landsat 7) with a collection that has 19 bands (Landsat 8).

    +
    // Merge Landsat collections.  
+var landsat78 = landsat7FiltMaskedRenamed  
+   .merge(landsat8FiltMasked.select(  
+       ['SR_B2', 'SR_B3', 'SR_B4', 'SR_B5', 'SR_B6', 'SR_B7']))  
+   .map(function(img) {       return img.toFloat();  
+   });  
+print('Merged collections', landsat78);
    +

    Now we have a collection with about 1000 images. Next, we will take the median of the values across the ImageCollection.

    +
    // Create Landsat 7 and 8 image composite and add to the Map.  
+var landsat78composite = landsat78.median().clip(country);  
+Map.addLayer(landsat78composite, visParams, 'L7 and L8 composite');

    Comparing the composite generated considering both Landsat 7 and 8 to the Landsat 8-only composite, it is evident that there is a reduction in the amount of data gaps in the final result (Fig. F4.3.9). The resulting Landsat 7 and 8 image composite still has data gaps due to the presence of clouds and Landsat 7’s SLC-off data. You can try setting the center of the map to the point with latitude 3.6023 and longitude −75.0741 to see the inset example of Fig. F4.3.9.

    -

    -

    Fig. F4.3.9 Landsat 8-only composite (left) and Landsat 7 and 8 composite (right) for 2019. Inset centered at latitude 3.6023, longitude −75.0741.

    +
    +
    +

    +

    Fig. F4.3.9 Landsat 8-only composite (left) and Landsat 7 and 8 composite (right) for 2019. Inset centered at latitude 3.6023, longitude −75.0741.

    +
    +
    @@ -1145,12 +1353,12 @@ Note
    -

    Code Checkpoint F43b. The book’s repository contains a script that shows what your code should look like at this point.

    +

    Code Checkpoint F43b. The book’s repository contains a script that shows what your code should look like at this point.

    -
    -

    9.2 Best-Available-Pixel Compositing Earth Engine Application

    +
    +

    9.3 Best-Available-Pixel Compositing Earth Engine Application

    This section presents an Earth Engine application that enables the generation of annual best-available-pixel (BAP) image composites by globally combining multiple Landsat sensors and images: GEE-BAP. Annual BAP image composites are generated by choosing optimal observations for each pixel from all available Landsat 5 TM, Landsat 7 ETM+, and Landsat 8 OLI imagery within a given year and within a given day range from a specified acquisition day of year, in addition to other constraints defined by the user. The data accessible via Earth Engine are from the USGS free and open archive of Landsat data. The Landsat images used are atmospherically corrected to surface reflectance values. Following White et al. (2014), a series of scoring functions ranks each pixel observation for (1) acquisition day of year, (2) cloud cover in the scene, (3) distance to clouds and cloud shadows, (4) presence of haze, and (5) acquisition sensor. Further information on the BAP image compositing approach can be found in Griffiths et al. (2013), and detailed information on tuning parameters can be found in White et al. (2014).

    @@ -1162,51 +1370,64 @@ Note
    -

    Code Checkpoint F43c. The book’s repository contains information about accessing the GEE-BAP interface and its related functions.

    +

    Code Checkpoint F43c. The book’s repository contains information about accessing the GEE-BAP interface and its related functions.

    -

    -

    Fig. F4.3.10 GEE-BAP user interface controls

    -

    Once you have loaded the GEE-BAP interface (Fig. F4.3.10) using the instructions in the Code Checkpoint, you will notice that it is divided into three sections: (1) Input/Output options, (2) Pixel scoring options, and (3) Advanced parameters. Users indicate the study area, the time period for generating annual BAP composites (i.e., start and end years), and the path to store the results in the Input/Output options. Users have three options to define the study area. The Draw study area option uses the Draw a shape and Draw a rectangle tools to define the area of interest. The Upload image template option utilizes an image template uploaded by the user in TIFF format. This option is well suited to generating BAP composites that match the projection, pixel size, and extent to existing raster datasets. The Work globally option generates BAP composites for the entire globe; note that when this option is selected, complete data download is not available due to the Earth’s size. With Start year and End year, users can indicate the beginning and end of the annual time series of BAP image composites to be generated. Multiple image composites are then generated—one composite for each year—resulting in a time series of annual composites. For each year, composites are uniquely generated utilizing images acquired on the days within the specified Date range. Produced BAP composites can be saved in the indicated (Path) Google Drive folder using the Tasks tab. Results are generated in a tiled, TIFF format, accompanied by a CSV file that indicates the parameters used to construct the composite.

    -

    As noted, GEE-BAP implements five pixel scoring functions: (1) target acquisition day of year and day range, (2) maximum cloud coverage per scene, (3) distance to clouds and cloud shadows, (4) atmospheric opacity, and (5) a penalty for images acquired under the Landsat 7 ETM+ SLC-off malfunction. By defining the Acquisition day of year and Day range, those candidate pixels acquired closer to a defined acquisition day of year are ranked higher. Note that pixels acquired outside the day range window are excluded from subsequent composite development. For example, if the target day of year is defined as “08-01” and the day range as “31,” only those pixels acquired between July 1 and August 31 are considered, and the ones acquired closer to August 1 will receive a higher score.

    -

    The scoring function Max cloud cover in scene indicates the maximum percentage of cloud cover in an image that will be accepted by the user in the BAP image compositing process. Defining a value of 70% implies that only those scenes with less than or equal to 70% cloud cover will be considered as a candidate for compositing.

    -

    The Distance to clouds and cloud shadows scoring function enables the user to exclude those pixels identified to contain clouds and shadows by the QA mask from the generated BAP, as well as decreasing a pixel’s score if the pixel is within a specified proximity of a cloud or cloud shadow.

    -

    The Atmospheric opacity scoring function ranks pixels based on their atmospheric opacity values, which are indicative of hazy imagery. Pixels with opacity values that exceed a defined haze expectation (Max opacity) are excluded. Pixels with opacity values lower than a defined value (Min opacity) get the maximum score. Pixels with values in between these limits are scored following the functions defined by Griffiths et al. (2013). This scoring function is available only for Landsat 5 TM and Landsat 7 ETM+ imagery, which provides the opacity attribute in the image metadata file.

    -

    Finally, there is a Landsat 7 ETM+ SLC-off penalty scoring function that de-emphasizes images acquired following the ETM+ SLC-off malfunction in 2003. The aim of this scoring element is to ensure that TM or OLI data, which do not have stripes, take precedence over ETM+ when using dates after the SLC failure. This allows users to avoid the inclusion of multiple discontinuous small portions of images being used to produce the BAP image composites, thus reducing the spatial variability of the spectral data. The penalty applied to SLC-off imagery is defined directly proportional to the overall score. A large score reduces the chance that SLC-off imagery will be used in the composite. A value of 1 prevents SLC-off imagery from being used.

    -

    By default, the GEE-BAP application produces image composites using all the visible bands. The Spectral index option enables the user to produce selected spectral indices from the resulting BAP image composites. Available spectral indices include: Normalized Difference Vegetation Index (NDVI, Fig. F4.3.11), Enhanced Vegetation Index (EVI), and Normalized Burn Ratio (NBR), as well as several indices derived from the Tasseled Cap transformation: Wetness (TCW), Greenness (TCG), Brightness (TCB), and Angle (TCA). Composited indices are able to be downloaded as well as viewed on the map.

    -

    -

    Fig. F4.3.11 Example of a global BAP image composite showing NDVI values generated using the GEE-BAP user interface

    +

    Fig. F4.3.10 GEE-BAP user interface controlsFig. F4.3.10 GEE-BAP user interface controlsFig. F4.3.10 GEE-BAP user interface controls

    +

    Once you have loaded the GEE-BAP interface (Fig. F4.3.10) using the instructions in the Code Checkpoint, you will notice that it is divided into three sections: (1) Input/Output options, (2) Pixel scoring options, and (3) Advanced parameters. Users indicate the study area, the time period for generating annual BAP composites (i.e., start and end years), and the path to store the results in the Input/Output options. Users have three options to define the study area. The Draw study area option uses the Draw a shape and Draw a rectangle tools to define the area of interest. The Upload image template option utilizes an image template uploaded by the user in TIFF format. This option is well suited to generating BAP composites that match the projection, pixel size, and extent to existing raster datasets. The Work globally option generates BAP composites for the entire globe; note that when this option is selected, complete data download is not available due to the Earth’s size. With Start year and End year, users can indicate the beginning and end of the annual time series of BAP image composites to be generated. Multiple image composites are then generated—one composite for each year—resulting in a time series of annual composites. For each year, composites are uniquely generated utilizing images acquired on the days within the specified Date range. Produced BAP composites can be saved in the indicated (Path) Google Drive folder using the Tasks tab. Results are generated in a tiled, TIFF format, accompanied by a CSV file that indicates the parameters used to construct the composite.

    +

    As noted, GEE-BAP implements five pixel scoring functions: (1) target acquisition day of year and day range, (2) maximum cloud coverage per scene, (3) distance to clouds and cloud shadows, (4) atmospheric opacity, and (5) a penalty for images acquired under the Landsat 7 ETM+ SLC-off malfunction. By defining the Acquisition day of year and Day range, those candidate pixels acquired closer to a defined acquisition day of year are ranked higher. Note that pixels acquired outside the day range window are excluded from subsequent composite development. For example, if the target day of year is defined as “08-01” and the day range as “31,” only those pixels acquired between July 1 and August 31 are considered, and the ones acquired closer to August 1 will receive a higher score.

    +

    The scoring function Max cloud cover in scene indicates the maximum percentage of cloud cover in an image that will be accepted by the user in the BAP image compositing process. Defining a value of 70% implies that only those scenes with less than or equal to 70% cloud cover will be considered as a candidate for compositing.

    +

    The Distance to clouds and cloud shadows scoring function enables the user to exclude those pixels identified to contain clouds and shadows by the QA mask from the generated BAP, as well as decreasing a pixel’s score if the pixel is within a specified proximity of a cloud or cloud shadow.

    +

    The Atmospheric opacity scoring function ranks pixels based on their atmospheric opacity values, which are indicative of hazy imagery. Pixels with opacity values that exceed a defined haze expectation (Max opacity) are excluded. Pixels with opacity values lower than a defined value (Min opacity) get the maximum score. Pixels with values in between these limits are scored following the functions defined by Griffiths et al. (2013). This scoring function is available only for Landsat 5 TM and Landsat 7 ETM+ imagery, which provides the opacity attribute in the image metadata file.

    +

    Finally, there is a Landsat 7 ETM+ SLC-off penalty scoring function that de-emphasizes images acquired following the ETM+ SLC-off malfunction in 2003. The aim of this scoring element is to ensure that TM or OLI data, which do not have stripes, take precedence over ETM+ when using dates after the SLC failure. This allows users to avoid the inclusion of multiple discontinuous small portions of images being used to produce the BAP image composites, thus reducing the spatial variability of the spectral data. The penalty applied to SLC-off imagery is defined directly proportional to the overall score. A large score reduces the chance that SLC-off imagery will be used in the composite. A value of 1 prevents SLC-off imagery from being used.

    +

    By default, the GEE-BAP application produces image composites using all the visible bands. The Spectral index option enables the user to produce selected spectral indices from the resulting BAP image composites. Available spectral indices include: Normalized Difference Vegetation Index (NDVI, Fig. F4.3.11), Enhanced Vegetation Index (EVI), and Normalized Burn Ratio (NBR), as well as several indices derived from the Tasseled Cap transformation: Wetness (TCW), Greenness (TCG), Brightness (TCB), and Angle (TCA). Composited indices are able to be downloaded as well as viewed on the map.

    +
    +
    +

    +

    Fig. F4.3.11 Example of a global BAP image composite showing NDVI values generated using the GEE-BAP user interface

    +
    +

    GEE-BAP functions can be accessed programmatically, including pixel scoring parameters, as well as BAP image compositing (BAP), de-spiking (despikeCollection), data-gap infilling (infill), and displaying (ShowCollection) functions. The following code sets the scoring parameter values, then generates and displays the compositing results (Fig. F4.3.12) for a BAP composite that is de-spiked, with data gaps infilled using temporal interpolation. Copy and paste the code below into a new script.

    -

    // Define required parameters.
    -var targetDay = ‘06-01’;
    -var daysRange = 75;
    -var cloudsTh = 70;
    -var SLCoffPenalty = 0.7;
    -var opacityScoreMin = 0.2;
    -var opacityScoreMax = 0.3;
    -var cloudDistMax = 1500;
    -var despikeTh = 0.65;
    -var despikeNbands = 3;
    -var startYear = 2015;
    -var endYear = 2017;

    -

    // Define study area.
    -var worldCountries = ee.FeatureCollection(‘USDOS/LSIB_SIMPLE/2017’);
    -var colombia = worldCountries.filter(ee.Filter.eq(‘country_na’,   ‘Colombia’));

    -

    // Load the bap library.
    -var library = require(‘users/sfrancini/bap:library’);

    -

    // Calculate BAP.
    -var BAPCS = library.BAP(null, targetDay, daysRange, cloudsTh,
    -   SLCoffPenalty, opacityScoreMin, opacityScoreMax, cloudDistMax);

    -

    // Despike the collection.
    -BAPCS = library.despikeCollection(despikeTh, despikeNbands, BAPCS,   1984, 2021, true);

    -

    // Infill datagaps.
    -BAPCS = library.infill(BAPCS, 1984, 2021, false, true);

    -

    // Visualize the image.
    -Map.centerObject(colombia, 5);
    -library.ShowCollection(BAPCS, startYear, endYear, colombia, false,   null);
    -library.AddSLider(startYear, endYear);

    -

    -

    Fig. F4.3.12 Outcome of the compositing code

    +
    // Define required parameters.  
+var targetDay = '06-01';  
+var daysRange = 75;  
+var cloudsTh = 70;  
+var SLCoffPenalty = 0.7;  
+var opacityScoreMin = 0.2;  
+var opacityScoreMax = 0.3;  
+var cloudDistMax = 1500;  
+var despikeTh = 0.65;  
+var despikeNbands = 3;  
+var startYear = 2015;  
+var endYear = 2017;  
+  
+// Define study area.  
+var worldCountries = ee.FeatureCollection('USDOS/LSIB_SIMPLE/2017');  
+var colombia = worldCountries.filter(ee.Filter.eq('country_na',   'Colombia'));  
+  
+// Load the bap library.  
+var library = require('users/sfrancini/bap:library');  
+  
+// Calculate BAP.  
+var BAPCS = library.BAP(null, targetDay, daysRange, cloudsTh,  
+   SLCoffPenalty, opacityScoreMin, opacityScoreMax, cloudDistMax);  
+  
+// Despike the collection.  
+BAPCS = library.despikeCollection(despikeTh, despikeNbands, BAPCS,   1984, 2021, true);  
+  
+// Infill datagaps.  
+BAPCS = library.infill(BAPCS, 1984, 2021, false, true);  
+  
+// Visualize the image.  
+Map.centerObject(colombia, 5);  
+library.ShowCollection(BAPCS, startYear, endYear, colombia, false,   null);  
+library.AddSLider(startYear, endYear);
    +
    +
    +

    +

    Fig. F4.3.12 Outcome of the compositing code

    +
    +
    @@ -1217,19 +1438,19 @@ Note
    -

    Code Checkpoint F43d. The book’s repository contains a script that shows what your code should look like at this point.

    +

    Code Checkpoint F43d. The book’s repository contains a script that shows what your code should look like at this point.

    Synthesis

    -

    Assignment 1. Create composites for other cloudy regions or less cloudy regions. For example, change the country variable to ‘Cambodia’ or ‘Mozambique’. Are more gaps present in the resulting composite? Can you change the compositing rules to improve this (using Acquisition day of year and Day range)? Different regions of the Earth have different cloud seasonal patterns, so the most appropriate date windows to acquire cloud-free composites will change depending on location. Also be aware that the larger the country, the longer it will take to generate the composite.

    -

    Assignment 2. Similarly, try creating composites for the wet and dry seasons of a region separately. Compare the two composites. Are some features brighter or darker? Is there evidence of drying of vegetation, such as related to leaf loss or reduction in herbaceous ground vegetation?

    +

    Assignment 1. Create composites for other cloudy regions or less cloudy regions. For example, change the country variable to ‘Cambodia’ or ‘Mozambique’. Are more gaps present in the resulting composite? Can you change the compositing rules to improve this (using Acquisition day of year and Day range)? Different regions of the Earth have different cloud seasonal patterns, so the most appropriate date windows to acquire cloud-free composites will change depending on location. Also be aware that the larger the country, the longer it will take to generate the composite.

    +

    Assignment 2. Similarly, try creating composites for the wet and dry seasons of a region separately. Compare the two composites. Are some features brighter or darker? Is there evidence of drying of vegetation, such as related to leaf loss or reduction in herbaceous ground vegetation?

    Assignment 3. Test different cloud threshold values and see if you can find an optimal threshold that balances data gaps against area coverage for your particular target date.

    Conclusion

    -

    We cannot monitor what we cannot see. Image compositing algorithms provide robust and transparent tools to address issues with clouds, cloud shadows, haze, and smoke in remotely sensed images derived from optical satellite data, and expand data availability for remote sensing applications. The tools and approaches described here should provide you with some useful strategies to aid in mitigating the presence of cloud cover in your data. Note that the quality of image outcomes is a function of the quality of cloud masking routines applied to the source data to generate the various flags that are used in the scoring functions described herein. Different compositing parameters can be used to represent a given location as a function of conditions that are present at a given point in time and the information needs of the end user. Tuning or optimization of compositing parameters is possible (and recommended) to ensure best capture of the physical conditions of interest.

    +

    We cannot monitor what we cannot see. Image compositing algorithms provide robust and transparent tools to address issues with clouds, cloud shadows, haze, and smoke in remotely sensed images derived from optical satellite data, and expand data availability for remote sensing applications. The tools and approaches described here should provide you with some useful strategies to aid in mitigating the presence of cloud cover in your data. Note that the quality of image outcomes is a function of the quality of cloud masking routines applied to the source data to generate the various flags that are used in the scoring functions described herein. Different compositing parameters can be used to represent a given location as a function of conditions that are present at a given point in time and the information needs of the end user. Tuning or optimization of compositing parameters is possible (and recommended) to ensure best capture of the physical conditions of interest.

    References

    @@ -1259,21 +1480,31 @@ Note

    10 Change Detection

    -

    ::: {.callout-tip} # Chapter Information

    +
    +
    +
    + +
    +
    +Chapter Information +
    +
    +

    Author

    Karis Tenneson, John Dilger, Crystal Wespestad, Brian Zutta, Andréa P Nicolau, Karen Dyson, Paula Paz

    Overview

    -

    This chapter introduces change detection mapping. It will teach you how to make a two-date land cover change map using image differencing and threshold-based classification. You will use what you have learned so far in this book to produce a map highlighting changes in the land cover between two time steps. You will first explore differences between the two images extracted from these time steps by creating a difference layer. You will then learn how to directly classify change based on the information in both of your images.

    +

    This chapter introduces change detection mapping. It will teach you how to make a two-date land cover change map using image differencing and threshold-based classification. You will use what you have learned so far in this book to produce a map highlighting changes in the land cover between two time steps. You will first explore differences between the two images extracted from these time steps by creating a difference layer. You will then learn how to directly classify change based on the information in both of your images.

    Learning Outcomes

    • Creating and exploring how to read a false-color cloud-free Landsat composite image
      • -
    • Calculating the Normalized Burn Ratio (NBR) index.  
      • -
    • Creating a two-image difference to help locate areas of change.
      • +
    • Calculating the Normalized Burn Ratio (NBR) index.
      +
      • +
    • Creating a two-image difference to help locate areas of change.
    • Producing a change map and classifying changes using thresholding.
    @@ -1284,9 +1515,11 @@ Note
  • Perform basic image analysis: select bands, compute indices, create masks (Part F2).
    • + +

    Introduction

    -

    Change detection is the process of assessing how landscape conditions are changing by looking at differences in images acquired at different times. This can be used to quantify changes in forest cover—such as those following a volcanic eruption, logging activity, or wildfire—or when crops are harvested (Fig. F4.4.1). For example, using time-series change detection methods, Hansen et al. (2013) quantified annual changes in forest loss and regrowth. Change detection mapping is important for observing, monitoring, and quantifying changes in landscapes over time. Key questions that can be answered using these techniques include identifying whether a change has occurred, measuring the area or the spatial extent of the region undergoing change, characterizing the nature of the change, and measuring the pattern (configuration or composition) of the change (MacLeod and Congalton 1998).

    +

    Change detection is the process of assessing how landscape conditions are changing by looking at differences in images acquired at different times. This can be used to quantify changes in forest cover—such as those following a volcanic eruption, logging activity, or wildfire—or when crops are harvested (Fig. F4.4.1). For example, using time-series change detection methods, Hansen et al. (2013) quantified annual changes in forest loss and regrowth. Change detection mapping is important for observing, monitoring, and quantifying changes in landscapes over time. Key questions that can be answered using these techniques include identifying whether a change has occurred, measuring the area or the spatial extent of the region undergoing change, characterizing the nature of the change, and measuring the pattern (configuration or composition) of the change (MacLeod and Congalton 1998).

    @@ -1302,60 +1535,72 @@ Note
    -

    -

    Fig. F4.4.1 Before and after images of (a) the eruption of Mount St. Helens in Washington State, USA, in 1980 (before, July 10, 1979; after, September 5, 1980); (b) the Camp Fire in California, USA, in 2018 (before, October 7, 2018; after, March 16, 2019); (c) illegal gold mining in the Madre de Dios region of Peru (before, March 31, 2001; after, August 22, 2020); and (d) shoreline changes in Incheon, South Korea (before, May 29, 1981; after, March 11, 2020)

    +
    +
    +

    +

    Fig. F4.4.1 Before and after images of (a) the eruption of Mount St. Helens in Washington State, USA, in 1980 (before, July 10, 1979; after, September 5, 1980); (b) the Camp Fire in California, USA, in 2018 (before, October 7, 2018; after, March 16, 2019); (c) illegal gold mining in the Madre de Dios region of Peru (before, March 31, 2001; after, August 22, 2020); and (d) shoreline changes in Incheon, South Korea (before, May 29, 1981; after, March 11, 2020)

    +
    +

    Many change detection techniques use the same basic premise: that most changes on the landscape result in spectral values that differ between pre-event and post-event images. The challenge can be to separate the real changes of interest—those due to activities on the landscape—from noise in the spectral signal, which can be caused by seasonal variation and phenology, image misregistration, clouds and shadows, radiometric inconsistencies, variability in illumination (e.g., sun angle, sensor position), and atmospheric effects.

    -

    Activities that result in pronounced changes in radiance values for a sufficiently long time period are easier to detect using remote sensing change detection techniques than are subtle or short-lived changes in landscape conditions. Mapping challenges can arise if the change event is short-lived, as these are difficult to capture using satellite instruments that only observe a location every several days. Other types of changes occur so slowly or are so vast that they are not easily detected until they are observed using satellite images gathered over a sufficiently long interval of time. Subtle changes that occur slowly on the landscape may be better suited to more computationally demanding methods, such as time-series analysis. Kennedy et al. (2009) provides a nice overview of the concepts and tradeoffs involved when designing landscape monitoring approaches. Additional summaries of change detection methods and recent advances include Singh (1989), Coppin et al. (2004), Lu et al. (2004), and Woodcock et al. (2020).

    -

    For land cover changes that occur abruptly over large areas on the landscape and are long-lived, a simple two-date image differencing approach is suitable. Two-date image differencing techniques are long-established methods for identifying changes that produce easily interpretable results (Singh 1989). The process typically involves four steps: (1) image selection and preprocessing; (2) data transformation, such as calculating the difference between indices of interest (e.g., the Normalized Difference Vegetation Index (NDVI)) in the pre-event and post-event images; (3) classifying the differenced image(s) using thresholding or supervised classification techniques; and (4) evaluation.

    -

    For the practicum, you will select pre-event and post-event image scenes and investigate the conditions in these images in a false-color composite display. Next, you will calculate the NBR index for each scene and create a difference image using the two NBR maps. Finally, you will apply a threshold to the difference image to establish categories of changed versus stable areas (Fig. F4.4.2).

    -

    -

    Fig. F4.4.2 Change detection workflow for this practicum

    +

    Activities that result in pronounced changes in radiance values for a sufficiently long time period are easier to detect using remote sensing change detection techniques than are subtle or short-lived changes in landscape conditions. Mapping challenges can arise if the change event is short-lived, as these are difficult to capture using satellite instruments that only observe a location every several days. Other types of changes occur so slowly or are so vast that they are not easily detected until they are observed using satellite images gathered over a sufficiently long interval of time. Subtle changes that occur slowly on the landscape may be better suited to more computationally demanding methods, such as time-series analysis. Kennedy et al. (2009) provides a nice overview of the concepts and tradeoffs involved when designing landscape monitoring approaches. Additional summaries of change detection methods and recent advances include Singh (1989), Coppin et al. (2004), Lu et al. (2004), and Woodcock et al. (2020).

    +

    For land cover changes that occur abruptly over large areas on the landscape and are long-lived, a simple two-date image differencing approach is suitable. Two-date image differencing techniques are long-established methods for identifying changes that produce easily interpretable results (Singh 1989). The process typically involves four steps: (1) image selection and preprocessing; (2) data transformation, such as calculating the difference between indices of interest (e.g., the Normalized Difference Vegetation Index (NDVI)) in the pre-event and post-event images; (3) classifying the differenced image(s) using thresholding or supervised classification techniques; and (4) evaluation.

    +

    For the practicum, you will select pre-event and post-event image scenes and investigate the conditions in these images in a false-color composite display. Next, you will calculate the NBR index for each scene and create a difference image using the two NBR maps. Finally, you will apply a threshold to the difference image to establish categories of changed versus stable areas (Fig. F4.4.2).

    +
    +
    +

    +

    Fig. F4.4.2 Change detection workflow for this practicum

    +
    +

    10.1 Preparing Imagery

    Before beginning a change detection workflow, image preprocessing is essential. The goal is to ensure that each pixel records the same type of measurement at the same location over time. These steps include multitemporal image registration and radiometric and atmospheric corrections, which are especially important. A lot of this work has been automated and already applied to the images that are available in Earth Engine. Image selection is also important. Selection considerations include finding images with low cloud cover and representing the same phenology (e.g., leaf-on or leaf-off).

    -

    The code in the block below accesses the USGS Landsat 8 Level 2, Collection 2, Tier 1 dataset and assigns it to the variable landsat8. To improve readability when working with the Landsat 8 ImageCollection, the code selects bands 2–7 and renames them to band names instead of band numbers.

    -

    var landsat8 = ee.ImageCollection(‘LANDSAT/LC08/C02/T1_L2’)
    -   .select(
    -       [‘SR_B2’, ‘SR_B3’, ‘SR_B4’, ‘SR_B5’, ‘SR_B6’, ‘SR_B7’],
    -       [‘blue’, ‘green’, ‘red’, ‘nir’, ‘swir1’, ‘swir2’]
    -   );

    -

    Next, you will split the Landsat 8 ImageCollection into two collections, one for each time period, and apply some filtering and sorting to get an image for each of two time periods. In this example, we know there are few clouds for the months of the analysis; if you’re working in a different area, you may need to apply some cloud masking or mosaicing techniques (see Chap. F4.3).

    -

    The code below does several things. First, it creates a new geometry variable to filter the geographic bounds of the image collections. Next, it creates a new variable for the pre-event image by (1) filtering the collection by the date range of interest (e.g., June 2013), (2) filtering the collection by the geometry, (3) sorting by cloud cover so the first image will have the least cloud cover, and (4) getting the first image from the collection.

    -

    Now repeat the previous step, but assign it to a post-event image variable and change the filter date to a period after the pre-event image’s date range (e.g., June 2020).  

    -

    var point = ee.Geometry.Point([-123.64, 42.96]);
    +

    The code in the block below accesses the USGS Landsat 8 Level 2, Collection 2, Tier 1 dataset and assigns it to the variable landsat8. To improve readability when working with the Landsat 8 ImageCollection, the code selects bands 2–7 and renames them to band names instead of band numbers.

    +

    var landsat8 = ee.ImageCollection(‘LANDSAT/LC08/C02/T1_L2’)
    +.select(
    +[‘SR_B2’, ‘SR_B3’, ‘SR_B4’, ‘SR_B5’, ‘SR_B6’, ‘SR_B7’],
    +[‘blue’, ‘green’, ‘red’, ‘nir’, ‘swir1’, ‘swir2’]
    +);

    +

    Next, you will split the Landsat 8 ImageCollection into two collections, one for each time period, and apply some filtering and sorting to get an image for each of two time periods. In this example, we know there are few clouds for the months of the analysis; if you’re working in a different area, you may need to apply some cloud masking or mosaicing techniques (see Chap. F4.3).

    +

    The code below does several things. First, it creates a new geometry variable to filter the geographic bounds of the image collections. Next, it creates a new variable for the pre-event image by (1) filtering the collection by the date range of interest (e.g., June 2013), (2) filtering the collection by the geometry, (3) sorting by cloud cover so the first image will have the least cloud cover, and (4) getting the first image from the collection.

    +

    Now repeat the previous step, but assign it to a post-event image variable and change the filter date to a period after the pre-event image’s date range (e.g., June 2020).

    +

    var point = ee.Geometry.Point([-123.64, 42.96]);
    Map.centerObject(point, 11);

    -

    var preImage = landsat8
    -   .filterBounds(point)
    -   .filterDate(‘2013-06-01’, ‘2013-06-30’)
    -   .sort(‘CLOUD_COVER’, true)
    -   .first();   var postImage = landsat8
    -   .filterBounds(point)
    -   .filterDate(‘2020-06-01’, ‘2020-06-30’)
    -   .sort(‘CLOUD_COVER’, true)
    -   .first();

    +

    var preImage = landsat8
    +.filterBounds(point)
    +.filterDate(‘2013-06-01’, ‘2013-06-30’)
    +.sort(‘CLOUD_COVER’, true)
    +.first(); var postImage = landsat8
    +.filterBounds(point)
    +.filterDate(‘2020-06-01’, ‘2020-06-30’)
    +.sort(‘CLOUD_COVER’, true)
    +.first();

    10.2 Creating False-Color Composites

    Before running any sort of change detection analysis, it is useful to first visualize your input images to get a sense of the landscape, visually inspect where changes might occur, and identify any problems in the inputs before moving further. As described in Chap. F1.1, false-color composites draw bands from multispectral sensors in the red, green, and blue channels in ways that are designed to illustrate contrast in imagery. Below, you will produce a false-color composite using SWIR-2 in the red channel, NIR in the green channel, and Red in the blue channel (Fig. F4.4.3).

    -

    Following the format in the code block below, first create a variable visParam to hold the display parameters, selecting the SWIR-2, NIR, and red bands, with values drawn that are between 7750 and 22200. Next, add the pre-event and post-event images to the map and click Run. Click and drag the opacity slider on the post-event image layer back and forth to view the changes between your two images.

    -

    var visParam = {   ‘bands’: [‘swir2’, ‘nir’, ‘red’],   ‘min’: 7750,   ‘max’: 22200
    +

    Following the format in the code block below, first create a variable visParam to hold the display parameters, selecting the SWIR-2, NIR, and red bands, with values drawn that are between 7750 and 22200. Next, add the pre-event and post-event images to the map and click Run. Click and drag the opacity slider on the post-event image layer back and forth to view the changes between your two images.

    +

    var visParam = { ‘bands’: [‘swir2’, ‘nir’, ‘red’], ‘min’: 7750, ‘max’: 22200
    };
    Map.addLayer(preImage, visParam, ‘pre’);
    Map.addLayer(postImage, visParam, ‘post’);

    -

    -

    Fig. F4.4.3 False-color composite using SWIR2, NIR, and red. Vegetation shows up vividly in the green channel due to vegetation being highly reflective in the NIR band. Shades of green can be indicative of vegetation density; water typically shows up as black to dark blue; and burned or barren areas show up as brown.

    +
    +
    +

    +

    Fig. F4.4.3 False-color composite using SWIR2, NIR, and red. Vegetation shows up vividly in the green channel due to vegetation being highly reflective in the NIR band. Shades of green can be indicative of vegetation density; water typically shows up as black to dark blue; and burned or barren areas show up as brown.

    +
    +

    10.3 Calculating NBR

    -

    The next step is data transformation, such as calculating NBR. The advantage of using these techniques is that the data, along with the noise inherent in the data, have been reduced in order to simplify a comparison between two images. Image differencing is done by subtracting the spectral value of the first-date image from that of the second-date image, pixel by pixel (Fig. F4.4.2). Two-date image differencing can be used with a single band or with spectral indices, depending on the application. Identifying the correct band or index to identify change and finding the correct thresholds to classify it are critical to producing meaningful results. Working with indices known to highlight the land cover conditions before and after a change event of interest is a good starting point. For example, the Normalized Difference Water Index would be good for mapping water level changes during flooding events; the NBR is good at detecting soil brightness; and the NDVI can be used for tracking changes in vegetation (although this index does saturate quickly). In some cases, using derived band combinations that have been customized to represent the phenomenon of interest is suggested, such as using the Normalized Difference Fraction Index to monitor forest degradation (see Chap. A3.4).

    -

    Examine changes to the landscape caused by fires using NBR, which measures the severity of fires using the equation (NIR − SWIR) / (NIR + SWIR). These bands were chosen because they respond most strongly to the specific changes in forests caused by fire. This type of equation, a difference of variables divided by their sum, is referred to as a normalized difference equation (see Chap. F2.0). The resulting value will always fall between −1 and 1. NBR is useful for determining whether a fire recently occurred and caused damage to the vegetation, but it is not designed to detect other types of land cover changes especially well.

    -

    First, calculate the NBR for each time period using the built-in normalized difference function. For Landsat 8, be sure to utilize the NIR and SWIR2 bands to calculate NBR. Then, rename each image band with the built-in rename function.

    -

    // Calculate NBR.
    -var nbrPre = preImage.normalizedDifference([‘nir’, ‘swir2’])
    -   .rename(‘nbr_pre’);
    -var nbrPost = postImage.normalizedDifference([‘nir’, ‘swir2’])
    -   .rename(‘nbr_post’);

    +

    The next step is data transformation, such as calculating NBR. The advantage of using these techniques is that the data, along with the noise inherent in the data, have been reduced in order to simplify a comparison between two images. Image differencing is done by subtracting the spectral value of the first-date image from that of the second-date image, pixel by pixel (Fig. F4.4.2). Two-date image differencing can be used with a single band or with spectral indices, depending on the application. Identifying the correct band or index to identify change and finding the correct thresholds to classify it are critical to producing meaningful results. Working with indices known to highlight the land cover conditions before and after a change event of interest is a good starting point. For example, the Normalized Difference Water Index would be good for mapping water level changes during flooding events; the NBR is good at detecting soil brightness; and the NDVI can be used for tracking changes in vegetation (although this index does saturate quickly). In some cases, using derived band combinations that have been customized to represent the phenomenon of interest is suggested, such as using the Normalized Difference Fraction Index to monitor forest degradation (see Chap. A3.4).

    +

    Examine changes to the landscape caused by fires using NBR, which measures the severity of fires using the equation (NIR − SWIR) / (NIR + SWIR). These bands were chosen because they respond most strongly to the specific changes in forests caused by fire. This type of equation, a difference of variables divided by their sum, is referred to as a normalized difference equation (see Chap. F2.0). The resulting value will always fall between −1 and 1. NBR is useful for determining whether a fire recently occurred and caused damage to the vegetation, but it is not designed to detect other types of land cover changes especially well.

    +

    First, calculate the NBR for each time period using the built-in normalized difference function. For Landsat 8, be sure to utilize the NIR and SWIR2 bands to calculate NBR. Then, rename each image band with the built-in rename function.

    +
    // Calculate NBR.  
+var nbrPre = preImage.normalizedDifference(['nir', 'swir2'])  
+   .rename('nbr_pre');  
+var nbrPost = postImage.normalizedDifference(['nir', 'swir2'])  
+   .rename('nbr_post');
    @@ -1366,50 +1611,59 @@ Note
    -

    Code Checkpoint F44a. The book’s repository contains a script that shows what your code should look like at this point.

    +

    Code Checkpoint F44a. The book’s repository contains a script that shows what your code should look like at this point.

    10.4 Single Date Transformation

    Next, we will examine the changes that have occurred, as seen when comparing two specific dates in time.

    -

    Subtract the pre-event image from the post-event image using the subtract function. Add the two-date change image to the map with the specialized Fabio Crameri batlow color ramp (Crameri et al. 2020). This color ramp is an example of a color combination specifically designed to be readable by colorblind and color-deficient viewers. Being cognizant of your cartographic choices is an important part of making a good change map.

    -

    // 2-date change.
    -var diff = nbrPost.subtract(nbrPre).rename(‘change’);

    -

    var palette = [   ‘011959’, ‘0E365E’, ‘1D5561’, ‘3E6C55’, ‘687B3E’,   ‘9B882E’, ‘D59448’, ‘F9A380’, ‘FDB7BD’, ‘FACCFA’
    -];
    -var visParams = {
    -   palette: palette,
    -   min: -0.2,
    -   max: 0.2
    -};
    -Map.addLayer(diff, visParams, ‘change’);

    -

    Question 1. Try to interpret the resulting image before reading on. What patterns of change can you identify? Can you find areas that look like vegetation loss or gain?

    -

    The color ramp has dark blues for the lowest values, greens and oranges in the midrange, and pink for the highest values. We used nbrPre subtracted from nbrPost to identify changes in each pixel. Since NBR values are higher when vegetation is present, areas that are negative in the change image will represent pixels that were higher in the nbrPre image than in the nbrPost image. Conversely, positive differences mean that an area gained vegetation (Fig. F4.4.4). 

    +

    Subtract the pre-event image from the post-event image using the subtract function. Add the two-date change image to the map with the specialized Fabio Crameri batlow color ramp (Crameri et al. 2020). This color ramp is an example of a color combination specifically designed to be readable by colorblind and color-deficient viewers. Being cognizant of your cartographic choices is an important part of making a good change map.

    +
    // 2-date change.  
+var diff = nbrPost.subtract(nbrPre).rename('change');  
+  
+var palette = [   '011959', '0E365E', '1D5561', '3E6C55', '687B3E',   '9B882E', 'D59448', 'F9A380', 'FDB7BD', 'FACCFA'  
+];  
+var visParams = {  
+   palette: palette,  
+   min: -0.2,  
+   max: 0.2  
+};  
+Map.addLayer(diff, visParams, 'change');
    +

    Question 1. Try to interpret the resulting image before reading on. What patterns of change can you identify? Can you find areas that look like vegetation loss or gain?

    +

    The color ramp has dark blues for the lowest values, greens and oranges in the midrange, and pink for the highest values. We used nbrPre subtracted from nbrPost to identify changes in each pixel. Since NBR values are higher when vegetation is present, areas that are negative in the change image will represent pixels that were higher in the nbrPre image than in the nbrPost image. Conversely, positive differences mean that an area gained vegetation (Fig. F4.4.4).

      -
    1.                                            b)                                           c)
      2. +
    2.                                        b)                                           c)
    -

    -

    Fig. F4.4.4 (a) Two-date NBR difference; (b) pre-event image (June 2013) false-color composite; (c) post-event image (June 2020) false-color composite. In the change map (a), areas on the lower range of values (blue) depict areas where vegetation has been negatively affected, and areas on the higher range of values (pink) depict areas where there has been vegetation gain; the green/orange areas have experienced little change. In the pre-event and post-event images (b and c), the green areas indicate vegetation, while the brown regions are barren ground.

    +
    +
    +

    +

    Fig. F4.4.4 (a) Two-date NBR difference; (b) pre-event image (June 2013) false-color composite; (c) post-event image (June 2020) false-color composite. In the change map (a), areas on the lower range of values (blue) depict areas where vegetation has been negatively affected, and areas on the higher range of values (pink) depict areas where there has been vegetation gain; the green/orange areas have experienced little change. In the pre-event and post-event images (b and c), the green areas indicate vegetation, while the brown regions are barren ground.

    +
    +

    10.5 Classifying Change

    -

    Once the images have been transformed and differenced to highlight areas undergoing change, the next step is image classification into a thematic map consisting of stable and change classes. This can be done rather simply by thresholding the change layer, or by using classification techniques such as machine learning algorithms. One challenge of working with simple thresholding of the difference layer is knowing how to select a suitable threshold to partition changed areas from stable classes. On the other hand, classification techniques using machine learning algorithms partition the landscape using examples of reference data that you provide to train the classifier. This may or may not yield better results, but does require additional work to collect reference data and train the classifier. In the end, resources, timing, and the patterns of the phenomenon you are trying to map will determine which approach is suitable—or perhaps the activity you are trying to track requires something more advanced, such as a time-series approach that uses more than two dates of imagery.

    +

    Once the images have been transformed and differenced to highlight areas undergoing change, the next step is image classification into a thematic map consisting of stable and change classes. This can be done rather simply by thresholding the change layer, or by using classification techniques such as machine learning algorithms. One challenge of working with simple thresholding of the difference layer is knowing how to select a suitable threshold to partition changed areas from stable classes. On the other hand, classification techniques using machine learning algorithms partition the landscape using examples of reference data that you provide to train the classifier. This may or may not yield better results, but does require additional work to collect reference data and train the classifier. In the end, resources, timing, and the patterns of the phenomenon you are trying to map will determine which approach is suitable—or perhaps the activity you are trying to track requires something more advanced, such as a time-series approach that uses more than two dates of imagery.

    For this chapter, we will classify our image into categories using a simple, manual thresholding method, meaning we will decide the optimal values for when a pixel will be considered change or no-change in the image. Finding the ideal value is a considerable task and will be unique to each use case and set of inputs (e.g., the threshold values for a SWIR2 single-band change would be different from the thresholds for NDVI). For a look at a more advanced method of thresholding, check out the thresholding methods in Chap. A2.3.

    -

    First, you will define two variables for the threshold values for gain and loss. Next, create a new image with a constant value of 0. This will be the basis of our classification. Reclassify the new image using the where function. Classify loss areas as 2 where the difference image is less than or equal to the loss threshold value. Reclassify gain areas to 1 where the difference image is greater than or equal to the gain threshold value. Finally, mask the image by itself and add the classified image to the map (Fig. F4.4.5). Note: It is not necessary to self-mask the image, and in many cases you might be just as interested in areas that did not change as you are in areas that did.

    -

    // Classify change
    -var thresholdGain = 0.10;
    -var thresholdLoss = -0.10;

    -

    var diffClassified = ee.Image(0);

    -

    diffClassified = diffClassified.where(diff.lte(thresholdLoss), 2);
    -diffClassified = diffClassified.where(diff.gte(thresholdGain), 1);

    -

    var changeVis = {
    -   palette: ‘fcffc8,2659eb,fa1373’,
    -   min: 0,
    -   max: 2
    -};

    -

    Map.addLayer(diffClassified.selfMask(),
    -   changeVis,   ‘change classified by threshold’);

    +

    First, you will define two variables for the threshold values for gain and loss. Next, create a new image with a constant value of 0. This will be the basis of our classification. Reclassify the new image using the where function. Classify loss areas as 2 where the difference image is less than or equal to the loss threshold value. Reclassify gain areas to 1 where the difference image is greater than or equal to the gain threshold value. Finally, mask the image by itself and add the classified image to the map (Fig. F4.4.5). Note: It is not necessary to self-mask the image, and in many cases you might be just as interested in areas that did not change as you are in areas that did.

    +
    // Classify change  
+var thresholdGain = 0.10;  
+var thresholdLoss = -0.10;  
+  
+var diffClassified = ee.Image(0);  
+  
+diffClassified = diffClassified.where(diff.lte(thresholdLoss), 2);  
+diffClassified = diffClassified.where(diff.gte(thresholdGain), 1);  
+  
+var changeVis = {  
+   palette: 'fcffc8,2659eb,fa1373',  
+   min: 0,  
+   max: 2  
+};  
+  
+Map.addLayer(diffClassified.selfMask(),  
+   changeVis,   'change classified by threshold');
    @@ -1417,9 +1671,8 @@ diffClassified = diffClassified.where(diff.gte(thresholdGain), 1);

    -

    -

    Fig. F4.4.5 (a) Change detection in timber forests of southern Oregon, including maps of the (left to right) pre-event false-color composite, post-event false-color composite, difference image, and classified change using NBR; (b) the same map types for an example of change caused by fire in southern Oregon. The false-color maps highlight vegetation in green and barren ground in brown. The difference images show NBR gain in pink to NBR loss in blue. The classified change images show NBR gain in blue and NBR loss in red.

    -

    Chapters F4.5 through F4.9 present more-advanced change detection algorithms that go beyond differencing and thresholding between two images, instead allowing you to analyze changes indicated across several images as a time series.

    +

    Fig. F4.4.5 (a) Change detection in timber forests of southern Oregon, including maps of the (left to right) pre-event false-color composite, post-event false-color composite, difference image, and classified change using NBR; (b) the same map types for an example of change caused by fire in southern Oregon. The false-color maps highlight vegetation in green and barren ground in brown. The difference images show NBR gain in pink to NBR loss in blue. The classified change images show NBR gain in blue and NBR loss in red.Fig. F4.4.5 (a) Change detection in timber forests of southern Oregon, including maps of the (left to right) pre-event false-color composite, post-event false-color composite, difference image, and classified change using NBR; (b) the same map types for an example of change caused by fire in southern Oregon. The false-color maps highlight vegetation in green and barren ground in brown. The difference images show NBR gain in pink to NBR loss in blue. The classified change images show NBR gain in blue and NBR loss in red.

    +

    Chapters F4.5 through F4.9 present more-advanced change detection algorithms that go beyond differencing and thresholding between two images, instead allowing you to analyze changes indicated across several images as a time series.

    @@ -1430,7 +1683,7 @@ Note
    -

    Code Checkpoint F44b. The book’s repository contains a script that shows what your code should look like at this point.

    +

    Code Checkpoint F44b. The book’s repository contains a script that shows what your code should look like at this point.

    @@ -1438,8 +1691,8 @@ Note

    Synthesis

    Evaluating any maps you create, including change detection maps, is essential to determining whether the method you have selected is appropriate for informing land management and decision-making (Stehman and Czaplewski 1998), or whether you need to iterate on the mapping process to improve the final results. Maps generally, and change maps specifically, will always have errors. This is due to a suite of factors, such as the geometric registration between images, the calibration between images, the data resolution (e.g., temporal, spectral, radiometric) compared to the characteristics of the activity of interest, the complexity of the landscape of the study region (topography, atmospheric conditions, etc.), and the classification techniques employed (Lu et al. 2004). This means that similar studies can present different, sometimes controversial, conclusions about landscape dynamics (e.g., Cohen et al. 2017). In order to be useful for decision-making, a change detection mapping effort should provide the user with an understanding of the strengths and weaknesses of the product, such as by presenting omission and commission error rates. The quantification of classification quality is presented in Chap. F2.2.

    Assignment 1. Try using a different index, such as NDVI or a Tasseled Cap Transformation, to run the change detection steps, and compare the results with those obtained from using NBR.

    -

    Assignment 2. Experiment with adjusting the thresholdLoss and thresholdGain values.

    -

    Assignment 3. Use what you have learned in the classification chapter (Chap. F2.1) to run a supervised classification on the difference layer (or layers, if you have created additional ones). Hint: To complete a supervised classification, you would need reference examples of both the stable and change classes of interest to train the classifier.

    +

    Assignment 2. Experiment with adjusting the thresholdLoss and thresholdGain values.

    +

    Assignment 3. Use what you have learned in the classification chapter (Chap. F2.1) to run a supervised classification on the difference layer (or layers, if you have created additional ones). Hint: To complete a supervised classification, you would need reference examples of both the stable and change classes of interest to train the classifier.

    Assignment 4. Think about how things like clouds and cloud shadows could affect the results of change detection. What do you think the two-date differencing method would pick up for images in the same year in different seasons?

    @@ -1456,17 +1709,26 @@ Note

    Kennedy RE, Townsend PA, Gross JE, et al (2009) Remote sensing change detection tools for natural resource managers: Understanding concepts and tradeoffs in the design of landscape monitoring projects. Remote Sens Environ 113:1382–1396. https://doi.org/10.1016/j.rse.2008.07.018

    Lu D, Mausel P, Brondízio E, Moran E (2004) Change detection techniques. Int J Remote Sens 25:2365–2401. https://doi.org/10.1080/0143116031000139863

    Macleod RD, Congalton RG (1998) A quantitative comparison of change-detection algorithms for monitoring eelgrass from remotely sensed data. Photogramm Eng Remote Sensing 64:207–216

    -

    Singh A (1989) Digital change detection techniques using remotely-sensed data. Int J  Remote Sens 10:989–1003. https://doi.org/10.1080/01431168908903939

    +

    Singh A (1989) Digital change detection techniques using remotely-sensed data. Int J Remote Sens 10:989–1003. https://doi.org/10.1080/01431168908903939

    Stehman SV, Czaplewski RL (1998) Design and analysis for thematic map accuracy assessment: Fundamental principles. Remote Sens Environ 64:331–344. https://doi.org/10.1016/S0034-4257(98)00010-8

    Woodcock CE, Loveland TR, Herold M, Bauer ME (2020) Transitioning from change detection to monitoring with remote sensing: A paradigm shift. Remote Sens Environ 238:111558. https://doi.org/10.1016/j.rse.2019.111558

    11 Interpreting Annual Time Series with LandTrendr

    -

    ::: {.callout-tip} # Chapter Information

    +
    +
    +
    + +
    +
    +Chapter Information +
    +
    +

    Author

    -

    Robert Kennedy, Justin Braaten, Peter Clary

    +

    Robert Kennedy, Justin Braaten, Peter Clary

    Overview

    @@ -1477,7 +1739,7 @@ Note
    • Evaluating yearly time-series spectral values to distinguish between true change and artifacts.
    • Recognizing disturbance and growth signals in the time series of annual spectral values for individual pixels.
      • -
    • Interpreting change segments and translating them to maps.
      • +
    • Interpreting change segments and translating them to maps.
    • Applying parameters in a graphical user interface to create disturbance maps in forests.
    @@ -1488,6 +1750,8 @@ Note
  • Interpret bands and indices in terms of land surface characteristics (Chap. F2.0).
    • + +

    Introduction

    Land surface change happens all the time, and satellite sensors witness it. If a spectral index is chosen to match the type of change being sought, surface change can be inferred from changes in spectral index values. Over time, the progression of spectral values witnessed in each pixel tells a story of the processes of change, such as growth and disturbance. Time-series algorithms are designed to leverage many observations of spectral values over time to isolate and describe changes of interest, while ignoring uninteresting change or noise.

    @@ -1503,114 +1767,148 @@ Note
    -

    Code Checkpoint F45a. The book’s repository contains information about accessing the LandTrendr interface.

    +

    Code Checkpoint F45a. The book’s repository contains information about accessing the LandTrendr interface.

    11.1 Pixel Time Series

    -

    When working with LandTrendr for the first time in your area, there are two questions you must address.  

    -

    First, is the change of interest detectable in the spectral reflectance record? If the change you are interested in does not leave a pattern in the spectral reflectance record, then an algorithm will not be able to find it.  

    +

    When working with LandTrendr for the first time in your area, there are two questions you must address.

    +

    First, is the change of interest detectable in the spectral reflectance record? If the change you are interested in does not leave a pattern in the spectral reflectance record, then an algorithm will not be able to find it.

    Second, can you identify fitting parameters that allow the algorithm to capture that record? Time series algorithms apply rules to a temporal sequence of spectral values in a pixel, and simplify the many observations into more digestible forms, such as the linear segments we will work with using LandTrendr. The algorithms that do the simplification are often guided by parameters that control the way the algorithm does its job.

    -

    The best way to begin assessing these questions is to look at the time series of individual pixels. In Earth Engine, open and run the script that generates the GUI we have developed to easily deploy the LandTrendr algorithms. Run the script, and you should see an interface that looks like the one shown in Fig. 4.5.1.

    -

    -

    Fig. 4.5.1 The LandTrendr GUI interface, with the control panel on the left, the  Map panel in the center, and the reporting panel on the right

    -

    The LandTrendr GUI consists of three panels: a control panel on the left, a reporting panel on the right, and a Map panel in the center. The control panel is where all of the functionality of the interface resides. There are several modules,and each is accessed by clicking on the double arrow to the right of the title. The  Map panel defaults to a location in Oregon but can be manually moved anywhere in the world. The reporting panel shows messages about how to use functions, as well as providing graphical outputs.

    -

    Next, expand the “Pixel Time Series Options” function. For now, simply use your mouse to click somewhere on the map. Wait a few seconds even though it looks like nothing is happening – be patient!! The GUI has sent information to Earth Engine to run the LandTrendr algorithms at the location you have clicked, and is waiting for the results. Eventually you should see a chart appear in the reporting panel on the right. Fig. 4.5.2 shows what one pixel looks like in an area where the forest burned and began regrowth. Your chart will probably look different.

    -

    -

    Fig. 4.5.2 A typical trajectory for a single pixel. The x-axis shows the year, the y-axis the spectral index value, and the title the index chosen. The gray line represents the original spectral values observed by Landsat, and the red line the result of the LandTrendr temporal segmentation algorithms.

    +

    The best way to begin assessing these questions is to look at the time series of individual pixels. In Earth Engine, open and run the script that generates the GUI we have developed to easily deploy the LandTrendr algorithms. Run the script, and you should see an interface that looks like the one shown in Fig. 4.5.1.

    +
    +
    +

    +

    Fig. 4.5.1 The LandTrendr GUI interface, with the control panel on the left, the Map panel in the center, and the reporting panel on the right

    +
    +
    +

    The LandTrendr GUI consists of three panels: a control panel on the left, a reporting panel on the right, and a Map panel in the center. The control panel is where all of the functionality of the interface resides. There are several modules,and each is accessed by clicking on the double arrow to the right of the title. The Map panel defaults to a location in Oregon but can be manually moved anywhere in the world. The reporting panel shows messages about how to use functions, as well as providing graphical outputs.

    +

    Next, expand the “Pixel Time Series Options” function. For now, simply use your mouse to click somewhere on the map. Wait a few seconds even though it looks like nothing is happening – be patient!! The GUI has sent information to Earth Engine to run the LandTrendr algorithms at the location you have clicked, and is waiting for the results. Eventually you should see a chart appear in the reporting panel on the right. Fig. 4.5.2 shows what one pixel looks like in an area where the forest burned and began regrowth. Your chart will probably look different.

    +
    +
    +

    +

    Fig. 4.5.2 A typical trajectory for a single pixel. The x-axis shows the year, the y-axis the spectral index value, and the title the index chosen. The gray line represents the original spectral values observed by Landsat, and the red line the result of the LandTrendr temporal segmentation algorithms.

    +
    +

    The key to success with the LandTrendr algorithm is interpreting these time series. First, let’s examine the components of the chart. The x-axis shows the year of observation. With LandTrendr, only one observation per year is used to describe the history of a pixel; later, we will cover how you control that value. The y-axis shows the spectral value of the index that is chosen. In the default mode, the Normalized Burn Ratio (as described in Chap. F4.4). Note that you also have the ability to pick more indices using the checkboxes on the control panel on the left. Note that we scale floating point (decimal) indices by 1000. Thus, an NBR value of 1.0 would be displayed as 1000.

    -

    In the chart area, the thick gray line represents the spectral values observed by the satellite for the period of the year selected for a single 30 m Landsat pixel at the location you have chosen. The red line is the output from the temporal segmentation that is the heart of the LandTrendr algorithms. The title of the chart shows the spectral index, as well as the root-mean-square error of the fit. 

    +

    In the chart area, the thick gray line represents the spectral values observed by the satellite for the period of the year selected for a single 30 m Landsat pixel at the location you have chosen. The red line is the output from the temporal segmentation that is the heart of the LandTrendr algorithms. The title of the chart shows the spectral index, as well as the root-mean-square error of the fit.

    To interpret the time series, first know which way is “up” and “down” for the spectral index you’re interested in. For the NBR, the index goes up in value when there is more vegetation and less soil in a pixel. It goes down when there is less vegetation. For vegetation disturbance monitoring, this is useful.

    -

    Next, translate that change into the changes of interest for the change processes you’re interested in. For conifer forest systems, the NBR is useful because it drops precipitously when a disturbance occurs, and it rises as vegetation grows.  

    -

    In the case of Fig. 4.5.2, we interpret the abrupt drop as a disturbance, and the subsequent rise of the index as regrowth or recovery (though not necessarily to the same type of vegetation).

    -

    -

    Fig. 4.5.3 For the trajectory in Fig. 4.5.2, we can identify a segment capturing disturbance based on its abrupt drop in the NBR index, and the subsequent vegetative recovery

    -

    Tip: LandTrendr is able to accept any index, and advanced users are welcome to use indices of their own design. An important consideration is knowing which direction indicates “recovery” and “disturbance” for the topic you are interested in. The algorithms favor detection of disturbance and can be controlled to constrain how quickly recovery is assumed to occur (see parameters below).  

    +

    Next, translate that change into the changes of interest for the change processes you’re interested in. For conifer forest systems, the NBR is useful because it drops precipitously when a disturbance occurs, and it rises as vegetation grows.

    +

    In the case of Fig. 4.5.2, we interpret the abrupt drop as a disturbance, and the subsequent rise of the index as regrowth or recovery (though not necessarily to the same type of vegetation).

    +
    +
    +

    +

    Fig. 4.5.3 For the trajectory in Fig. 4.5.2, we can identify a segment capturing disturbance based on its abrupt drop in the NBR index, and the subsequent vegetative recovery

    +
    +
    +

    Tip: LandTrendr is able to accept any index, and advanced users are welcome to use indices of their own design. An important consideration is knowing which direction indicates “recovery” and “disturbance” for the topic you are interested in. The algorithms favor detection of disturbance and can be controlled to constrain how quickly recovery is assumed to occur (see parameters below).

    For LandTrendr to have any hope of finding the change of interest, that change must be manifested in the gray line showing the original spectral values. If you know that some process is occurring and it is not evident in the gray line, what can you do?

    One option is to change the index. Any single index is simply one view of the larger spectral space of the Landsat Thematic Mapper sensors. The change you are interested in may cause spectral change in a different direction than that captured with some indices. Try choosing different indices from the list. If you click on different checkboxes and re-submit the pixel, the fits for all of the different indices will appear.

    -

    Another option is to change the date range. LandTrendr uses one value per year, but the value that is chosen can be controlled by the user. It’s possible that the change of interest is better identified in some seasons than others. We use a medoid image compositing approach, which picks the best single observation each year from a date range of images in an ImageCollection. In the GUI, you can change the date range of imagery used for compositing in the Image Collection portion of the LandTrendr Options menu (Fig. F4.5.4).  

    -

    -

    Fig. 4.5.4 The LandTrendr options menu. Users control the year and date range in the Image Collection section, the index used for temporal segmentation in the middle section, and the parameters controlling the temporal segmentation in the bottom section

    -

    Change the Start Date and End Date to find a time of year when the distinction between cover conditions before and during the change process of interest is greatest.

    -

    There are other considerations to keep in mind. First, seasonality of vegetation, water, or snow often can affect the signal of the change of interest. And because we use an ImageCollection that spans a range of dates, it’s best to choose a date range where there is not likely to be a substantial change in vegetative state from the beginning to the end of the date range. Clouds can be a factor too. Some seasons will have more cloudiness, which can make it difficult to find good images. Often with optical sensors, we are constrained to working with periods where clouds are less prevalent, or using wide date ranges to provide many opportunities for a pixel to be cloud-free.

    +

    Another option is to change the date range. LandTrendr uses one value per year, but the value that is chosen can be controlled by the user. It’s possible that the change of interest is better identified in some seasons than others. We use a medoid image compositing approach, which picks the best single observation each year from a date range of images in an ImageCollection. In the GUI, you can change the date range of imagery used for compositing in the Image Collection portion of the LandTrendr Options menu (Fig. F4.5.4).

    +
    +
    +

    +

    Fig. 4.5.4 The LandTrendr options menu. Users control the year and date range in the Image Collection section, the index used for temporal segmentation in the middle section, and the parameters controlling the temporal segmentation in the bottom section

    +
    +
    +

    Change the Start Date and End Date to find a time of year when the distinction between cover conditions before and during the change process of interest is greatest.

    +

    There are other considerations to keep in mind. First, seasonality of vegetation, water, or snow often can affect the signal of the change of interest. And because we use an ImageCollection that spans a range of dates, it’s best to choose a date range where there is not likely to be a substantial change in vegetative state from the beginning to the end of the date range. Clouds can be a factor too. Some seasons will have more cloudiness, which can make it difficult to find good images. Often with optical sensors, we are constrained to working with periods where clouds are less prevalent, or using wide date ranges to provide many opportunities for a pixel to be cloud-free.

    It is possible that no combination of index or data range is sensitive to the change of interest. If that is the case, there are two options: try using a different sensor and change detection technique, or accept that the change is not discernible. This can often occur if the change of interest occupies a small portion of a given 30 m by 30 m Landsat pixel, or if the spectral manifestation of the change is so subtle that it is not spectrally separable from non-changed pixels

    -

    Even if you as a human can identify the change of interest in the spectral trajectory of the gray line, an algorithm may not be able to similarly track it. To give the algorithm a fighting chance, you need to explore whether different fitting parameters could be used to match the red fitted line with the gray source image line.  

    -

    The overall fitting process includes steps to reduce noise and best identify the underlying signal. The temporal segmentation algorithms are controlled by fitting parameters that are described in detail in Kennedy et al. (2010). You adjust these parameters using the Fitting Parameters block of the LandTrendr Options menu. Below is a brief overview of what values are often useful, but these will likely change as you use different spectral indices.

    -

    First, the minimum observations needed criterion is used to evaluate whether a given trajectory has enough unfiltered (i.e., clear observation) years to run the fitting. We suggest leaving this at the default of 6.

    -

    The segmentation begins with a noise-dampening step to remove spikes that could be caused by unfiltered clouds or shadows. The spike threshold parameter controls the degree of filtering. A value of 1.0 corresponds to no filtering, and lower values corresponding to more severe filtering. We suggest leaving this at 0.9; if changed, a range from 0.7 to 1.0 is appropriate.

    -

    The next step is finding vertices. This begins with the start and end year as vertex years, progressively adding candidate vertex years based on deviation from linear fits. To avoid getting an overabundance of vertex years initially found using this method, we suggest leaving the vertex count overshoot at a value of 3. A second set of algorithms uses deflection angle to cull back this overabundance to a set number of maximum candidate vertex years.

    -

    That number of vertex years is controlled by the max_segments parameter. As a general rule, your number of segments should be no more than one-third of the total number of likely yearly observations. The years of these vertices (X-values) are then passed to the model-building step. Assuming you are using at least 30 years of the archive, and your area has reasonable availability of images, a value of 8 is a good starting point.

    -

    In the model-building step, straight-line segments are built by fitting Y-values (spectral values) for the periods defined by the vertex years (X-values). The process moves from left to right—early years to late years. Regressions of each subsequent segment are connected to the end of the prior segment. Regressions are also constrained to prevent unrealistic recovery after disturbance, as controlled by the recovery threshold parameter. A lower value indicates greater constraint: a value of 1.0 means the constraint is turned off; a value of 0.25 means that segments that fully recover in faster than four years (4 = 1/0.25) are not permitted. Note: This parameter has strong control on the fitting, and is one of the first to explore when testing parameters.  Additionally, the preventOneYearRecovery will disallow fits that have one-year-duration recovery segments.  This may be useful to prevent overfitting of noisy data in environments where such quick vegetative recovery is not ecologically realistic.

    -

    Once a model of the maximum number of segments is found, successively simpler models are made by iteratively removing the least informative vertex. Each model is scored using a pseudo-f statistic, which penalizes models with more segments, to create a pseudo p-value for each model. The p-value threshold parameter is used to identify all fits that are deemed good enough. Start with a value of 0.05, but check to see if the fitted line appears to capture the salient shape and features of the gray source trajectory.  If you see temporal patterns in the gray line that are likely not noise (based on your understanding of the system under study), consider switching the p-value threshold to 0.10 or even 0.15. 

    +

    Even if you as a human can identify the change of interest in the spectral trajectory of the gray line, an algorithm may not be able to similarly track it. To give the algorithm a fighting chance, you need to explore whether different fitting parameters could be used to match the red fitted line with the gray source image line.

    +

    The overall fitting process includes steps to reduce noise and best identify the underlying signal. The temporal segmentation algorithms are controlled by fitting parameters that are described in detail in Kennedy et al. (2010). You adjust these parameters using the Fitting Parameters block of the LandTrendr Options menu. Below is a brief overview of what values are often useful, but these will likely change as you use different spectral indices.

    +

    First, the minimum observations needed criterion is used to evaluate whether a given trajectory has enough unfiltered (i.e., clear observation) years to run the fitting. We suggest leaving this at the default of 6.

    +

    The segmentation begins with a noise-dampening step to remove spikes that could be caused by unfiltered clouds or shadows. The spike threshold parameter controls the degree of filtering. A value of 1.0 corresponds to no filtering, and lower values corresponding to more severe filtering. We suggest leaving this at 0.9; if changed, a range from 0.7 to 1.0 is appropriate.

    +

    The next step is finding vertices. This begins with the start and end year as vertex years, progressively adding candidate vertex years based on deviation from linear fits. To avoid getting an overabundance of vertex years initially found using this method, we suggest leaving the vertex count overshoot at a value of 3. A second set of algorithms uses deflection angle to cull back this overabundance to a set number of maximum candidate vertex years.

    +

    That number of vertex years is controlled by the max_segments parameter. As a general rule, your number of segments should be no more than one-third of the total number of likely yearly observations. The years of these vertices (X-values) are then passed to the model-building step. Assuming you are using at least 30 years of the archive, and your area has reasonable availability of images, a value of 8 is a good starting point.

    +

    In the model-building step, straight-line segments are built by fitting Y-values (spectral values) for the periods defined by the vertex years (X-values). The process moves from left to right—early years to late years. Regressions of each subsequent segment are connected to the end of the prior segment. Regressions are also constrained to prevent unrealistic recovery after disturbance, as controlled by the recovery threshold parameter. A lower value indicates greater constraint: a value of 1.0 means the constraint is turned off; a value of 0.25 means that segments that fully recover in faster than four years (4 = 1/0.25) are not permitted. Note: This parameter has strong control on the fitting, and is one of the first to explore when testing parameters. Additionally, the preventOneYearRecovery will disallow fits that have one-year-duration recovery segments. This may be useful to prevent overfitting of noisy data in environments where such quick vegetative recovery is not ecologically realistic.

    +

    Once a model of the maximum number of segments is found, successively simpler models are made by iteratively removing the least informative vertex. Each model is scored using a pseudo-f statistic, which penalizes models with more segments, to create a pseudo p-value for each model. The p-value threshold parameter is used to identify all fits that are deemed good enough. Start with a value of 0.05, but check to see if the fitted line appears to capture the salient shape and features of the gray source trajectory. If you see temporal patterns in the gray line that are likely not noise (based on your understanding of the system under study), consider switching the p-value threshold to 0.10 or even 0.15.

    Note: because of temporal autocorrelation, these cannot be interpreted as true f- and p-values, but rather as relative scalars to distinguish goodness of fit among models. If no good models can be found using these criteria based on the p-value parameter set by the user, a second approach is used to solve for the Y-value of all vertex years simultaneously. If no good model is found, then a straight-line mean value model is used.

    -

    From the models that pass the p-value threshold, one is chosen as the final fit. It may be the one with the lowest p-value. However, an adjustment is made to allow more complicated models (those with more segments) to be picked even if their p-value is within a defined proportion of the best-scoring model. That proportion is set by the best model proportion parameter. As an example, a best model proportion value of 0.75 would allow a more complicated model to be chosen if its score were greater than 75% that of the best model.

    +

    From the models that pass the p-value threshold, one is chosen as the final fit. It may be the one with the lowest p-value. However, an adjustment is made to allow more complicated models (those with more segments) to be picked even if their p-value is within a defined proportion of the best-scoring model. That proportion is set by the best model proportion parameter. As an example, a best model proportion value of 0.75 would allow a more complicated model to be chosen if its score were greater than 75% that of the best model.

    11.2 Translating Pixels to Maps

    -

    Although the full time series is the best description of each pixel’s “life history,” we typically are interested in the behavior of all of the pixels in our study area. It would be both inefficient to manually visualize all of them and ineffective to try to summarize areas and locations. Thus, we seek to make maps.

    -

    There are three post-processing steps to convert a segmented trajectory to a map. First, we identify segments of interest; if we are interested in disturbance, we find segments whose spectral change indicates loss. Second, we filter out segments of that type that do not meet criteria of interest. For example, very low magnitude disturbances can occur when the algorithm mistakenly finds a pattern in the random noise of the signal, and thus we do not want to include it. Third, we extract from the segment of interest something about its character to map on a pixel-by-pixel basis: its start year, duration, spectral value, or the value of the spectral change.

    -

    Theory: We’ll start with a single pixel to learn how to Interpret a disturbance pixel time series in terms of the dominant disturbance segment. For the disturbance time series we have used in figures above, we can identify the key parameters of the segment associated with the disturbance. For the example above, we have extracted the actual NBR values of the fitted time series and noted them in a table (Fig. 4.5.5). This is not part of the GUI – it is simply used here to work through the concepts.

    -

    -

    Fig. 4.5.5 Tracking actual values of fitted trajectories to learn how we focus on quantification of disturbance. Because we know that the NBR index drops when vegetation is lost and soil exposure is increased, we know that a precipitous drop suggests an abrupt loss of vegetation. Although some early segments show very subtle change, only the segment between vertex 4 and 5 shows large-magnitude vegetation loss.

    +

    Although the full time series is the best description of each pixel’s “life history,” we typically are interested in the behavior of all of the pixels in our study area. It would be both inefficient to manually visualize all of them and ineffective to try to summarize areas and locations. Thus, we seek to make maps.

    +

    There are three post-processing steps to convert a segmented trajectory to a map. First, we identify segments of interest; if we are interested in disturbance, we find segments whose spectral change indicates loss. Second, we filter out segments of that type that do not meet criteria of interest. For example, very low magnitude disturbances can occur when the algorithm mistakenly finds a pattern in the random noise of the signal, and thus we do not want to include it. Third, we extract from the segment of interest something about its character to map on a pixel-by-pixel basis: its start year, duration, spectral value, or the value of the spectral change.

    +

    Theory: We’ll start with a single pixel to learn how to Interpret a disturbance pixel time series in terms of the dominant disturbance segment. For the disturbance time series we have used in figures above, we can identify the key parameters of the segment associated with the disturbance. For the example above, we have extracted the actual NBR values of the fitted time series and noted them in a table (Fig. 4.5.5). This is not part of the GUI – it is simply used here to work through the concepts.

    +
    +
    +

    +

    Fig. 4.5.5 Tracking actual values of fitted trajectories to learn how we focus on quantification of disturbance. Because we know that the NBR index drops when vegetation is lost and soil exposure is increased, we know that a precipitous drop suggests an abrupt loss of vegetation. Although some early segments show very subtle change, only the segment between vertex 4 and 5 shows large-magnitude vegetation loss.

    +
    +

    From the table shown in Fig. 4.5.5, we can infer several key things about this pixel:

      -
    • It was likely disturbed between 2006 and 2007. This is because the NBR value drops precipitously in the segment bounded by vertices (breakpoints) in 2006 and 2007.  
      • +
    • It was likely disturbed between 2006 and 2007. This is because the NBR value drops precipitously in the segment bounded by vertices (breakpoints) in 2006 and 2007.
      +
    • The magnitude of spectral change was large: 1175 scaled NBR units out of a possible range of 2000 scaled units.
    • There were small drops in NBR earlier, which may indicate some subtle loss of vegetation over a long period in the pixel. These drops, however, would need to be explored in a separate analysis because of their subtle nature.
    • The main disturbance had a disturbance duration of just one year. This abruptness combined with the high magnitude suggests a major vegetative disturbance such as a harvest or a fire.
    • The disturbance was then followed by recovery of vegetation, but not to the level before the disturbance. Note: Ecologists will recognize the growth signal as one of succession, or active revegetation by human intervention.
    -

    Following the three post-processing steps noted in the introduction to this section, to map the year of disturbance for this pixel we would first identify the potential disturbance segments as those with negative NBR. Then we would hone in on the disturbance of interest by filtering out potential disturbance segments that are not abrupt and/or of small magnitude. This would leave only the high-magnitude, short-duration segment. For that segment, the first year that we have evidence of disturbance is the first year after the start of the segment. The segment starts in 2006, which means that 2007 is the first year we have such evidence. Thus, we would assign 2007 to this pixel.

    +

    Following the three post-processing steps noted in the introduction to this section, to map the year of disturbance for this pixel we would first identify the potential disturbance segments as those with negative NBR. Then we would hone in on the disturbance of interest by filtering out potential disturbance segments that are not abrupt and/or of small magnitude. This would leave only the high-magnitude, short-duration segment. For that segment, the first year that we have evidence of disturbance is the first year after the start of the segment. The segment starts in 2006, which means that 2007 is the first year we have such evidence. Thus, we would assign 2007 to this pixel.

    If we wanted to map the magnitude of the disturbance, we would follow the same first two steps, but then report for the pixel value the magnitude difference between the starting and ending segment.

    -

    The LandTrendr GUI provides a set of tools to easily apply the same logic rules to all pixels of interest and create maps. Click on the Change Filter Options menu. The interface shown in Fig. 4.5.6 appears.

    -

    -

    Fig. 4.5.6 The menu used to post-process disturbance trajectories into maps. Select vegetation change type and sort to hone in on the segment type of interest, then check boxes to apply selective filters to eliminate uninteresting changes.  

    -

    The first two sections are used to identify the segments of interest.  

    -

    Select Vegetation Change Type offers the options of gain or loss, which refer to gain or loss of vegetation, with disturbance assumed to be related to loss of vegetation. Note: Advanced users can look in the landtrendr.js library in the “calcindex” function to add new indices with gain and loss defined as they choose. The underlying algorithm is built to find disturbance in indices that increase when disturbance occurs, so indices such as NBR or NDVI need to be multiplied by (−1) before being fed to the LandTrendr algorithm. This is handled in the calcIndex function.

    -

    Select Vegetation Change Sort offers various options that allow you to choose the segment of interest based on timing or duration. By default, the greatest magnitude disturbance is chosen.

    -

    Each filter (magnitude, duration, etc.) is used to further winnow the possible segments of interest. All other filters are applied at the pixel scale, but Filter by MMU is applied to groups of pixels based on a given minimum mapping unit (MMU). Once all other filters have been defined, some pixels are flagged as being of interest and others are not. The MMU filter looks to see how many connected pixels have been flagged as occurring in the same year, and omits groups smaller in pixel count than the number indicated here (which defaults to 11 pixels, or approximately 1 hectare).  

    -

    If you’re following along and making changes, or if you’re just using the default location and parameters, click the Add Filtered Disturbance Imagery to add this to the map. You should see something like Fig. 4.5.7.

    -

    -

    Fig. 4.5.7 The basic output from a disturbance mapping exercise  

    +

    The LandTrendr GUI provides a set of tools to easily apply the same logic rules to all pixels of interest and create maps. Click on the Change Filter Options menu. The interface shown in Fig. 4.5.6 appears.

    +
    +
    +

    +

    Fig. 4.5.6 The menu used to post-process disturbance trajectories into maps. Select vegetation change type and sort to hone in on the segment type of interest, then check boxes to apply selective filters to eliminate uninteresting changes.

    +
    +
    +

    The first two sections are used to identify the segments of interest.

    +

    Select Vegetation Change Type offers the options of gain or loss, which refer to gain or loss of vegetation, with disturbance assumed to be related to loss of vegetation. Note: Advanced users can look in the landtrendr.js library in the “calcindex” function to add new indices with gain and loss defined as they choose. The underlying algorithm is built to find disturbance in indices that increase when disturbance occurs, so indices such as NBR or NDVI need to be multiplied by (−1) before being fed to the LandTrendr algorithm. This is handled in the calcIndex function.

    +

    Select Vegetation Change Sort offers various options that allow you to choose the segment of interest based on timing or duration. By default, the greatest magnitude disturbance is chosen.

    +

    Each filter (magnitude, duration, etc.) is used to further winnow the possible segments of interest. All other filters are applied at the pixel scale, but Filter by MMU is applied to groups of pixels based on a given minimum mapping unit (MMU). Once all other filters have been defined, some pixels are flagged as being of interest and others are not. The MMU filter looks to see how many connected pixels have been flagged as occurring in the same year, and omits groups smaller in pixel count than the number indicated here (which defaults to 11 pixels, or approximately 1 hectare).

    +

    If you’re following along and making changes, or if you’re just using the default location and parameters, click the Add Filtered Disturbance Imagery to add this to the map. You should see something like Fig. 4.5.7.

    +
    +
    +

    +

    Fig. 4.5.7 The basic output from a disturbance mapping exercise

    +
    +

    There are multiple layers of disturbance added to the map. Use the map layers checkboxes to change which is shown. Magnitude of disturbance, for example, is a map of the delta change between beginning and endpoints of the segments (Fig. 4.5.8).

    -

    -

    Fig. 4.5.8 Magnitude of change for the same area

    +
    +
    +

    +

    Fig. 4.5.8 Magnitude of change for the same area

    +
    +

    Synthesis

    -

    In this chapter, you have learned how to work with annual time series to interpret regions of interest. Looking at annual snapshots of the landscape provides three key benefits: (1) the ability to view your area of interest without the clouds and noise that typically obscure single-day views; (2) gauge the amount by which the noise-dampened signal still varies from year to year in response to large-scale forcing mechanisms; and (3) the ability to view the response of landscapes as they recover, sometimes slowly, from disturbance.

    +

    In this chapter, you have learned how to work with annual time series to interpret regions of interest. Looking at annual snapshots of the landscape provides three key benefits: (1) the ability to view your area of interest without the clouds and noise that typically obscure single-day views; (2) gauge the amount by which the noise-dampened signal still varies from year to year in response to large-scale forcing mechanisms; and (3) the ability to view the response of landscapes as they recover, sometimes slowly, from disturbance.

    To learn more about LandTrendr, see the assignments below.

    -

    Assignment 1. Find your own change processes of interest. First, navigate the map (zooming and dragging) to an area of the world where you are interested in a change process, and the spectral index that would capture it. Make sure the UI control panel is open to the Pixel Time-Series Options section. Next, click on the map in areas where you know change has occurred, and observe the spectral trajectories in the charts. Then, describe whether the change of interest is detectable in the spectral reflectance record, and what are its characteristics in different parts of the study area. .

    -

    Assignment 2: Find a pixel in your area of interest that shows a distinctive disturbance process, as you define it for your topic of interest. Adjust date ranges, parameters, etc. using the steps outlined in Section 1 above, and then answer these questions:

    +

    Assignment 1. Find your own change processes of interest. First, navigate the map (zooming and dragging) to an area of the world where you are interested in a change process, and the spectral index that would capture it. Make sure the UI control panel is open to the Pixel Time-Series Options section. Next, click on the map in areas where you know change has occurred, and observe the spectral trajectories in the charts. Then, describe whether the change of interest is detectable in the spectral reflectance record, and what are its characteristics in different parts of the study area. .

    +

    Assignment 2: Find a pixel in your area of interest that shows a distinctive disturbance process, as you define it for your topic of interest. Adjust date ranges, parameters, etc. using the steps outlined in Section 1 above, and then answer these questions:

      -
    • Question 1. Which index and date range did you use?
      • -
    • Question 2. Did you need to change fitting parameters to make the algorithm find the disturbance? If so, which ones, and why?
      • -
    • Question 3. How do you know this is a disturbance?
      • +
    • Question 1. Which index and date range did you use?
      • +
    • Question 2. Did you need to change fitting parameters to make the algorithm find the disturbance? If so, which ones, and why?
      • +
    • Question 3. How do you know this is a disturbance?
    -

    Assignment 3. Switch the control panel in the GUI to Change Filter Options, and use the guidance in Section 2 to set parameters and make disturbance maps.  

    +

    Assignment 3. Switch the control panel in the GUI to Change Filter Options, and use the guidance in Section 2 to set parameters and make disturbance maps.

      -
    • Question 4. Do the disturbance year and magnitude as mapped in the image match with what you would expect from the trajectory itself?
      • -
    • Question 5. Can you change some of the filters to create a map where your disturbance process is not mapped? If so, what did you change?  
      • -
    • Question 6. Can you change filters to create a map that includes a different disturbance process, perhaps subtler, longer duration, etc.? Find a pixel and use the “Pixel time series” plotter to look at the time series of those processes.
      • +
    • Question 4. Do the disturbance year and magnitude as mapped in the image match with what you would expect from the trajectory itself?
      • +
    • Question 5. Can you change some of the filters to create a map where your disturbance process is not mapped? If so, what did you change?
      +
      • +
    • Question 6. Can you change filters to create a map that includes a different disturbance process, perhaps subtler, longer duration, etc.? Find a pixel and use the “Pixel time series” plotter to look at the time series of those processes.
    -

    Assignment 4:  Return to the Pixel Time-Series Options section of the control panel, and navigate to a pixel in your area of interest that you believe would show a distinctive recovery or growth process, as you define it for your topic of interest. You may want to modify the index, parameters, etc. as covered in Section 1 to adequately capture the growth process with the fitted trajectories.

    +

    Assignment 4: Return to the Pixel Time-Series Options section of the control panel, and navigate to a pixel in your area of interest that you believe would show a distinctive recovery or growth process, as you define it for your topic of interest. You may want to modify the index, parameters, etc. as covered in Section 1 to adequately capture the growth process with the fitted trajectories.

      -
    • Question 7. Did you use the same spectral index? If not, why?
      • -
    • Question 8. Were the fitting parameters the same as those for disturbance? If not, what did you change, and why?
      • -
    • Question 9. What evidence do you have that this is a vegetative growth signal?
      • +
    • Question 7. Did you use the same spectral index? If not, why?
      • +
    • Question 8. Were the fitting parameters the same as those for disturbance? If not, what did you change, and why?
      • +
    • Question 9. What evidence do you have that this is a vegetative growth signal?
    -

    Assignment 5. For vegetation gain mapping, switch the control panel back to Change Filter Options and use the guidance in Section 2 to set parameters, etc. to make maps of growth.

    +

    Assignment 5. For vegetation gain mapping, switch the control panel back to Change Filter Options and use the guidance in Section 2 to set parameters, etc. to make maps of growth.

      -
    • Question 10. For the pixel or pixels you found for Assignment 3, does the year and magnitude as mapped in the “gain” image match with what you would expect from the trajectory itself?
      • -
    • Question 11. Compare what the map looks like when you run it with and without the MMU filter. What differences do you see?
      • -
    • Question 12. Try changing the recovery duration filter to a very high number (perhaps the full length of your archive) and to a very low number (say, one or two years). What differences do you see? 
      • +
    • Question 10. For the pixel or pixels you found for Assignment 3, does the year and magnitude as mapped in the “gain” image match with what you would expect from the trajectory itself?
      • +
    • Question 11. Compare what the map looks like when you run it with and without the MMU filter. What differences do you see?
      • +
    • Question 12. Try changing the recovery duration filter to a very high number (perhaps the full length of your archive) and to a very low number (say, one or two years). What differences do you see?

    Conclusion

    This exercise provides a baseline sense of how the LandTrendr algorithm works. The key points are learning how to interpret change in spectral values in terms of the processes occurring on the ground, and then translating those into maps.

    -

    You can export the images you’ve made here using Download Options. Links to materials are available in the chapter checkpoints and LandTrendr documentation about both the GUI and the script-based versions of the algorithm. In particular, there are scripts that handle different components of the fitting and mapping process, and that allow you to keep track of the fitting and image selection criteria.

    +

    You can export the images you’ve made here using Download Options. Links to materials are available in the chapter checkpoints and LandTrendr documentation about both the GUI and the script-based versions of the algorithm. In particular, there are scripts that handle different components of the fitting and mapping process, and that allow you to keep track of the fitting and image selection criteria.

    References

    @@ -1620,14 +1918,23 @@ Note

    12 Fitting Functions to Time Series

    -

    ::: {.callout-tip} # Chapter Information

    +
    +
    +
    + +
    +
    +Chapter Information +
    +
    +

    Author

    Andréa Puzzi Nicolau, Karen Dyson, Biplov Bhandari, David Saah, Nicholas Clinton

    Overview

    -

    The purpose of this chapter is to establish a foundation for time-series analysis of remotely sensed data, which is typically arranged as an ordered stack of images. You will be introduced to the concepts of graphing time series, using linear modeling to detrend time series, and fitting harmonic models to time-series data. At the completion of this chapter, you will be able to perform analysis of multi-temporal data for determining trend and seasonality on a per-pixel basis.

    +

    The purpose of this chapter is to establish a foundation for time-series analysis of remotely sensed data, which is typically arranged as an ordered stack of images. You will be introduced to the concepts of graphing time series, using linear modeling to detrend time series, and fitting harmonic models to time-series data. At the completion of this chapter, you will be able to perform analysis of multi-temporal data for determining trend and seasonality on a per-pixel basis.

    Learning Outcomes

    @@ -1642,85 +1949,103 @@ Note
    • Import images and image collections, filter, and visualize (Part F1).
    • Perform basic image analysis: select bands, compute indices, create masks (Part F2).
      • -
    • ​​Create a graph using ui.Chart (Chap. F1.3).
      • -
    • Use normalizedDifference to calculate vegetation indices (Chap. F2.0).
      • -
    • Write a function and map it over an ImageCollection (Chap. F4.0).
      • +
    • ​​Create a graph using ui.Chart (Chap. F1.3).
      • +
    • Use normalizedDifference to calculate vegetation indices (Chap. F2.0).
      • +
    • Write a function and map it over an ImageCollection (Chap. F4.0).
    • Mask cloud, cloud shadow, snow/ice, and other undesired pixels (Chap. F4.3).
    +
    +

    Introduction

    -

    Many natural and man-made phenomena exhibit important annual, interannual, or longer-term trends that recur—that is, they occur at roughly regular intervals. Examples include seasonality in leaf patterns in deciduous forests and seasonal crop growth patterns. Over time, indices such as the Normalized Difference Vegetation Index (NDVI) will show regular increases (e.g., leaf-on, crop growth) and decreases (e.g., leaf-off, crop senescence), and typically have a long-term, if noisy, trend such as a gradual increase in NDVI value as an area recovers from a disturbance.

    -

    Earth Engine supports the ability to do complex linear and non-linear regressions of values in each pixel of a study area. Simple linear regressions of indices can reveal linear trends that can span multiple years. Meanwhile, harmonic terms can be used to fit a sine-wave-like curve. Once you have the ability to fit these functions to time series, you can answer many important questions. For example, you can define vegetation dynamics over multiple time scales, identify phenology and track changes year to year, and identify deviations from the expected patterns (Bradley et al. 2007, Bullock et al. 2020). There are multiple applications for these analyses. For example, algorithms to detect deviations from the expected pattern can be used to identify disturbance events, including deforestation and forest degradation (Bullock et al. 2020).

    -

    If you have not already done so, be sure to add the book’s code repository to the Code Editor by entering https://code.earthengine.google.com/?accept_repo=projects/gee-edu/book into your browser. The book’s scripts will then be available in the script manager panel.

    +

    Many natural and man-made phenomena exhibit important annual, interannual, or longer-term trends that recur—that is, they occur at roughly regular intervals. Examples include seasonality in leaf patterns in deciduous forests and seasonal crop growth patterns. Over time, indices such as the Normalized Difference Vegetation Index (NDVI) will show regular increases (e.g., leaf-on, crop growth) and decreases (e.g., leaf-off, crop senescence), and typically have a long-term, if noisy, trend such as a gradual increase in NDVI value as an area recovers from a disturbance.

    +

    Earth Engine supports the ability to do complex linear and non-linear regressions of values in each pixel of a study area. Simple linear regressions of indices can reveal linear trends that can span multiple years. Meanwhile, harmonic terms can be used to fit a sine-wave-like curve. Once you have the ability to fit these functions to time series, you can answer many important questions. For example, you can define vegetation dynamics over multiple time scales, identify phenology and track changes year to year, and identify deviations from the expected patterns (Bradley et al. 2007, Bullock et al. 2020). There are multiple applications for these analyses. For example, algorithms to detect deviations from the expected pattern can be used to identify disturbance events, including deforestation and forest degradation (Bullock et al. 2020).

    +

    If you have not already done so, be sure to add the book’s code repository to the Code Editor by entering https://code.earthengine.google.com/?accept_repo=projects/gee-edu/book into your browser. The book’s scripts will then be available in the script manager panel.

    12.1 Multi-Temporal Data in Earth Engine

    -

    As explained in Chaps. F4.0 and F4.1, a time series in Earth Engine is typically represented as an ImageCollection. Because of image overlaps, cloud treatments, and filtering choices, an ImageCollection can have any of the following complex characteristics:

    +

    As explained in Chaps. F4.0 and F4.1, a time series in Earth Engine is typically represented as an ImageCollection. Because of image overlaps, cloud treatments, and filtering choices, an ImageCollection can have any of the following complex characteristics:

    • At each pixel, there might be a distinct number of observations taken from a unique set of dates.
    • The size (length) of the time series can vary across pixels.
    • Data may be missing in any pixel at any point in the sequence (e.g., due to cloud masking).
    -

    The use of multi-temporal data in Earth Engine introduces two mind-bending concepts, which we will describe below.

    -

    Per-pixel curve fitting. As you have likely encountered in many settings, a function can be fit through a series of values. In the most familiar example, a function of the form y = mx + b can represent a linear trend in data of all kinds. Fitting a straight “curve” with linear regression techniques involves estimating m and b for a set of x and y values. In a time series, x typically represents time, while y values represent observations at specific times. This chapter introduces how to estimate m and b for computed indices through time to model a potential linear trend in a time series. We then demonstrate how to fit a sinusoidal wave, which is useful for modeling rising and falling values, such as NDVI over a growing season. What can be particularly mind-bending in this setting is the fact that when Earth Engine is asked to estimate values across a large area, it will fit a function in every pixel of the study area. Each pixel, then, has its own m and b values, determined by the number of observations in that pixel, the observed values, and the dates for which they were observed.

    -

    Higher-dimension band values: array images. That more complex conception of the potential information contained in a single pixel can be represented in a higher-order Earth Engine structure: the array image. As you will encounter in this lab, it is possible for a single pixel in a single band of a single image to contain more than one value. If you choose to implement an array image, a single pixel might contain a one-dimensional vector of numbers, perhaps holding the slope and intercept values resulting from a linear regression, for example. Other examples, outside the scope of this chapter but used in the next chapter, might employ a two-dimensional matrix of values for each pixel within a single band of an image. Higher-order dimensions are available, as well as array image manipulations borrowed from the world of matrix algebra. Additionally, there are functions to move between the multidimensional array image structure and the more familiar, more easily displayed, simple Image type. Some of these array image functions were encountered in Chap. F3.1, but with less explanatory context.

    -

    First, we will give some very basic notation (Fig. F4.6.1). A scalar pixel at time t is given by pt, and a pixel vector by pt. A variable with a “hat” represents an estimated value: in this context, p̂t is the estimated pixel value at time t. A time series is a collection of pixel values, usually sorted chronologically: {pt; t = t0…tN}, where t might be in any units, t0 is the smallest, and tN is the largest such t in the series.

    -

    -

    Fig. F4.6.1 Time series representation of pixel p 

    +

    The use of multi-temporal data in Earth Engine introduces two mind-bending concepts, which we will describe below.

    +

    Per-pixel curve fitting. As you have likely encountered in many settings, a function can be fit through a series of values. In the most familiar example, a function of the form y = mx + b can represent a linear trend in data of all kinds. Fitting a straight “curve” with linear regression techniques involves estimating m and b for a set of x and y values. In a time series, x typically represents time, while y values represent observations at specific times. This chapter introduces how to estimate m and b for computed indices through time to model a potential linear trend in a time series. We then demonstrate how to fit a sinusoidal wave, which is useful for modeling rising and falling values, such as NDVI over a growing season. What can be particularly mind-bending in this setting is the fact that when Earth Engine is asked to estimate values across a large area, it will fit a function in every pixel of the study area. Each pixel, then, has its own m and b values, determined by the number of observations in that pixel, the observed values, and the dates for which they were observed.

    +

    Higher-dimension band values: array images. That more complex conception of the potential information contained in a single pixel can be represented in a higher-order Earth Engine structure: the array image. As you will encounter in this lab, it is possible for a single pixel in a single band of a single image to contain more than one value. If you choose to implement an array image, a single pixel might contain a one-dimensional vector of numbers, perhaps holding the slope and intercept values resulting from a linear regression, for example. Other examples, outside the scope of this chapter but used in the next chapter, might employ a two-dimensional matrix of values for each pixel within a single band of an image. Higher-order dimensions are available, as well as array image manipulations borrowed from the world of matrix algebra. Additionally, there are functions to move between the multidimensional array image structure and the more familiar, more easily displayed, simple Image type. Some of these array image functions were encountered in Chap. F3.1, but with less explanatory context.

    +

    First, we will give some very basic notation (Fig. F4.6.1). A scalar pixel at time t is given by pt, and a pixel vector by pt. A variable with a “hat” represents an estimated value: in this context, p̂t is the estimated pixel value at time t. A time series is a collection of pixel values, usually sorted chronologically: {pt; t = t0…tN}, where t might be in any units, t0 is the smallest, and tN is the largest such t in the series.

    +
    +
    +

    +

    Fig. F4.6.1 Time series representation of pixel p

    +
    +

    12.2 Data Preparation and Preprocessing

    -

    The first step in analysis of time-series data is to import data of interest and plot it at an interesting location. We will work with the USGS Landsat 8 Level 2, Collection 2, Tier 1 ImageCollection and a cloud-masking function (Chap. F4.3), scale the image values, and add variables of interest to the collection as bands. Copy and paste the code below to filter the Landsat 8 collection to a point of interest over California (variable roi) and specific dates, and to apply the defined function. The variables of interest added by the function are: (1) NDVI (Chap. F2.0), (2) a time variable that is the difference between the image’s current year and the year 1970 (a start point), and (3) a constant variable with value 1.

    -

    ///////////////////// Sections 1 & 2 /////////////////////////////

    -

    // Define function to mask clouds, scale, and add variables
    -// (NDVI, time and a constant) to Landsat 8 imagery.
    -function maskScaleAndAddVariable(image) {   // Bit 0 - Fill   // Bit 1 - Dilated Cloud   // Bit 2 - Cirrus   // Bit 3 - Cloud   // Bit 4 - Cloud Shadow   var qaMask = image.select(‘QA_PIXEL’).bitwiseAnd(parseInt(‘11111’,       2)).eq(0);   var saturationMask = image.select(‘QA_RADSAT’).eq(0);   // Apply the scaling factors to the appropriate bands.   var opticalBands = image.select(‘SR_B.’).multiply(0.0000275).add(-       0.2);   var thermalBands = image.select(’ST_B.*‘).multiply(0.00341802)
    -       .add(149.0);   // Replace the original bands with the scaled ones and apply the masks.   var img = image.addBands(opticalBands, null, true)
    -       .addBands(thermalBands, null, true)
    -       .updateMask(qaMask)
    -       .updateMask(saturationMask);   var imgScaled = image.addBands(img, null, true);   // Now we start to add variables of interest.   // Compute time in fractional years since the epoch.   var date = ee.Date(image.get(’system:time_start’));   var years = date.difference(ee.Date(‘1970-01-01’), ‘year’);   // Return the image with the added bands.   return imgScaled       // Add an NDVI band.       .addBands(imgScaled.normalizedDifference([‘SR_B5’, ‘SR_B4’])
    -           .rename(‘NDVI’))       // Add a time band.       .addBands(ee.Image(years).rename(‘t’))
    -       .float()       // Add a constant band.       .addBands(ee.Image.constant(1));
    -}

    -

    // Import point of interest over California, USA.
    -var roi = ee.Geometry.Point([-121.059, 37.9242]);

    -

    // Import the USGS Landsat 8 Level 2, Collection 2, Tier 1 image collection),
    -// filter, mask clouds, scale, and add variables.
    -var landsat8sr = ee.ImageCollection(‘LANDSAT/LC08/C02/T1_L2’)
    -   .filterBounds(roi)
    -   .filterDate(‘2013-01-01’, ‘2018-01-01’)
    -   .map(maskScaleAndAddVariable);

    -

    // Set map center over the ROI.
    -Map.centerObject(roi, 6);

    -

    Next, to visualize the NDVI at the point of interest over time, copy and paste the code below to print a chart of the time series (Chap. F1.3) at the location of interest (Fig. F4.6.2).

    -

    // Plot a time series of NDVI at a single location.
    -var landsat8Chart = ui.Chart.image.series(landsat8sr.select(‘NDVI’), roi)
    -   .setChartType(‘ScatterChart’)
    -   .setOptions({
    -       title: ‘Landsat 8 NDVI time series at ROI’,
    -       lineWidth: 1,
    -       pointSize: 3,
    -   });
    -print(landsat8Chart);

    -

    -

    Fig. F4.6.2 Time series representation of pixel p 

    -

    We can add a linear trend line to our chart using the trendlines parameters in the setOptions function for image series charts. Copy and paste the code below to print the same chart but with a linear trend line plotted (Fig. F4.6.3). In the next section, you will learn how to estimate linear trends over time.

    -

    // Plot a time series of NDVI with a linear trend line
    -// at a single location.
    -var landsat8ChartTL = ui.Chart.image.series(landsat8sr.select(‘NDVI’), roi)
    -   .setChartType(‘ScatterChart’)
    -   .setOptions({
    -       title: ‘Landsat 8 NDVI time series at ROI’,
    -       trendlines: {           0: {
    -               color: ‘CC0000’           }
    -       },
    -       lineWidth: 1,
    -       pointSize: 3,
    -   });
    -print(landsat8ChartTL);

    -

    -

    Fig. F4.6.3 Time series representation of pixel p with the trend line in red

    +

    The first step in analysis of time-series data is to import data of interest and plot it at an interesting location. We will work with the USGS Landsat 8 Level 2, Collection 2, Tier 1 ImageCollection and a cloud-masking function (Chap. F4.3), scale the image values, and add variables of interest to the collection as bands. Copy and paste the code below to filter the Landsat 8 collection to a point of interest over California (variable roi) and specific dates, and to apply the defined function. The variables of interest added by the function are: (1) NDVI (Chap. F2.0), (2) a time variable that is the difference between the image’s current year and the year 1970 (a start point), and (3) a constant variable with value 1.

    +
    ///////////////////// Sections 1 & 2 /////////////////////////////  
+  
+// Define function to mask clouds, scale, and add variables  
+// (NDVI, time and a constant) to Landsat 8 imagery.  
+function maskScaleAndAddVariable(image) {   // Bit 0 - Fill   // Bit 1 - Dilated Cloud   // Bit 2 - Cirrus   // Bit 3 - Cloud   // Bit 4 - Cloud Shadow   var qaMask = image.select('QA_PIXEL').bitwiseAnd(parseInt('11111',       2)).eq(0);   var saturationMask = image.select('QA_RADSAT').eq(0);   // Apply the scaling factors to the appropriate bands.   var opticalBands = image.select('SR_B.').multiply(0.0000275).add(-       0.2);   var thermalBands = image.select('ST_B.*').multiply(0.00341802)  
+       .add(149.0);   // Replace the original bands with the scaled ones and apply the masks.   var img = image.addBands(opticalBands, null, true)  
+       .addBands(thermalBands, null, true)  
+       .updateMask(qaMask)  
+       .updateMask(saturationMask);   var imgScaled = image.addBands(img, null, true);   // Now we start to add variables of interest.   // Compute time in fractional years since the epoch.   var date = ee.Date(image.get('system:time_start'));   var years = date.difference(ee.Date('1970-01-01'), 'year');   // Return the image with the added bands.   return imgScaled       // Add an NDVI band.       .addBands(imgScaled.normalizedDifference(['SR_B5', 'SR_B4'])  
+           .rename('NDVI'))       // Add a time band.       .addBands(ee.Image(years).rename('t'))  
+       .float()       // Add a constant band.       .addBands(ee.Image.constant(1));  
+}  
+  
+// Import point of interest over California, USA.  
+var roi = ee.Geometry.Point([-121.059, 37.9242]);  
+  
+// Import the USGS Landsat 8 Level 2, Collection 2, Tier 1 image collection),  
+// filter, mask clouds, scale, and add variables.  
+var landsat8sr = ee.ImageCollection('LANDSAT/LC08/C02/T1_L2')  
+   .filterBounds(roi)  
+   .filterDate('2013-01-01', '2018-01-01')  
+   .map(maskScaleAndAddVariable);  
+  
+// Set map center over the ROI.  
+Map.centerObject(roi, 6);
    +

    Next, to visualize the NDVI at the point of interest over time, copy and paste the code below to print a chart of the time series (Chap. F1.3) at the location of interest (Fig. F4.6.2).

    +
    // Plot a time series of NDVI at a single location.  
+var landsat8Chart = ui.Chart.image.series(landsat8sr.select('NDVI'), roi)  
+   .setChartType('ScatterChart')  
+   .setOptions({  
+       title: 'Landsat 8 NDVI time series at ROI',  
+       lineWidth: 1,  
+       pointSize: 3,  
+   });  
+print(landsat8Chart);
    +
    +
    +

    +

    Fig. F4.6.2 Time series representation of pixel p

    +
    +
    +

    We can add a linear trend line to our chart using the trendlines parameters in the setOptions function for image series charts. Copy and paste the code below to print the same chart but with a linear trend line plotted (Fig. F4.6.3). In the next section, you will learn how to estimate linear trends over time.

    +
    // Plot a time series of NDVI with a linear trend line  
+// at a single location.  
+var landsat8ChartTL = ui.Chart.image.series(landsat8sr.select('NDVI'), roi)  
+   .setChartType('ScatterChart')  
+   .setOptions({  
+       title: 'Landsat 8 NDVI time series at ROI',  
+       trendlines: {           0: {  
+               color: 'CC0000'           }  
+       },  
+       lineWidth: 1,  
+       pointSize: 3,  
+   });  
+print(landsat8ChartTL);
    +
    +
    +

    +

    Fig. F4.6.3 Time series representation of pixel p with the trend line in red

    +
    +

    Now that we have plotted and visualized the data, lots of interesting analyses can be done to the time series by harnessing Earth Engine tools for fitting curves through this data. We will see a couple of examples in the following sections.

    @@ -1732,56 +2057,69 @@ Note
    -

    Code Checkpoint F46a. The book’s repository contains a script that shows what your code should look like at this point.

    +

    Code Checkpoint F46a. The book’s repository contains a script that shows what your code should look like at this point.

    -

    12.3 Estimating Linear Trend Over Time

    +

    12.3 Estimating Linear Trend Over Time

    Time series datasets may contain not only trends but also seasonality, both of which may need to be removed prior to modeling. Trends and seasonality can result in a varying mean and a varying variance over time, both of which define a time series as non-stationary. Stationary datasets, on the other hand, have a stable mean and variance, and are therefore much easier to model.

    -

    Consider the following linear model, where et is a random error:

    -

    pt = β0 + β1t + et                                                        (Eq. F4.6.1)

    +

    Consider the following linear model, where et is a random error:

    +

    pt = β0 + β1t + et (Eq. F4.6.1)

    This is the model behind the trend line added to the chart created in the previous section (Fig. F4.6.3). Identifying trends at different scales is a big topic, with many approaches being used (e.g., differencing, modeling).

    -

    Removing unwanted to uninteresting trends for a given problem is often a first step to understanding complex patterns in time series. There are several approaches to remove trends. Here, we will remove the linear trend that is evident in the data shown in Fig. F4.6.3 using Earth Engine’s built-in tools for regression modeling. This approach is a useful, straightforward way to detrend data in time series (Shumway and Stoffer 2019). Here, the goal is to discover the values of the β’s in Eq. F4.6.1 for each pixel.

    -

    Copy and paste code below into the Code Editor, adding it to the end of the script from the previous section. Running this code will fit this trend model to the Landsat-based NDVI series using ordinary least squares, using the linearRegression reducer (Chap. F3.0).

    -

    ///////////////////// Section 3 /////////////////////////////

    -

    // List of the independent variable names
    -var independents = ee.List([‘constant’, ‘t’]);

    -

    // Name of the dependent variable.
    -var dependent = ee.String(‘NDVI’);

    -

    // Compute a linear trend.  This will have two bands: ‘residuals’ and
    -// a 2x1 (Array Image) band called ‘coefficients’.
    -// (Columns are for dependent variables)
    -var trend = landsat8sr.select(independents.add(dependent))
    -   .reduce(ee.Reducer.linearRegression(independents.length(), 1));
    -Map.addLayer(trend, {}, ‘trend array image’);

    -

    // Flatten the coefficients into a 2-band image.
    -var coefficients = trend.select(‘coefficients’)   // Get rid of extra dimensions and convert back to a regular image   .arrayProject([0])
    -   .arrayFlatten([independents]);
    -Map.addLayer(coefficients, {}, ‘coefficients image’);

    -

    If you click over a point using the Inspector tab, you will see the pixel values for the array image (coefficients “t” and “constant”, and residuals) and two-band image (coefficients “t” and “constant”) (Fig. F4.6.4).

    -

    -

    Fig. F4.6.4 Pixel values of array image and coefficients image

    -

    Now, copy and paste the code below to use the model to detrend the original NDVI time series and plot the time series chart with the trendlines parameter (Fig. F4.6.5).

    -

    // Compute a detrended series.
    -var detrended = landsat8sr.map(function(image) {   return image.select(dependent).subtract(
    -           image.select(independents).multiply(coefficients)
    -           .reduce(‘sum’))
    -       .rename(dependent)
    -       .copyProperties(image, [‘system:time_start’]);
    -});

    -

    // Plot the detrended results.
    -var detrendedChart = ui.Chart.image.series(detrended, roi, null, 30)
    -   .setOptions({
    -       title: ‘Detrended Landsat time series at ROI’,
    -       lineWidth: 1,
    -       pointSize: 3,
    -       trendlines: {           0: {
    -               color: ‘CC0000’           }
    -       },
    -   });print(detrendedChart);

    -

    -

    Fig. F4.6.5 Detrended NDVI time series

    +

    Removing unwanted to uninteresting trends for a given problem is often a first step to understanding complex patterns in time series. There are several approaches to remove trends. Here, we will remove the linear trend that is evident in the data shown in Fig. F4.6.3 using Earth Engine’s built-in tools for regression modeling. This approach is a useful, straightforward way to detrend data in time series (Shumway and Stoffer 2019). Here, the goal is to discover the values of the β’s in Eq. F4.6.1 for each pixel.

    +

    Copy and paste code below into the Code Editor, adding it to the end of the script from the previous section. Running this code will fit this trend model to the Landsat-based NDVI series using ordinary least squares, using the linearRegression reducer (Chap. F3.0).

    +
    ///////////////////// Section 3 /////////////////////////////  
+  
+// List of the independent variable names  
+var independents = ee.List(['constant', 't']);  
+  
+// Name of the dependent variable.  
+var dependent = ee.String('NDVI');  
+  
+// Compute a linear trend.  This will have two bands: 'residuals' and  
+// a 2x1 (Array Image) band called 'coefficients'.  
+// (Columns are for dependent variables)  
+var trend = landsat8sr.select(independents.add(dependent))  
+   .reduce(ee.Reducer.linearRegression(independents.length(), 1));  
+Map.addLayer(trend, {}, 'trend array image');  
+  
+// Flatten the coefficients into a 2-band image.  
+var coefficients = trend.select('coefficients')   // Get rid of extra dimensions and convert back to a regular image   .arrayProject([0])  
+   .arrayFlatten([independents]);  
+Map.addLayer(coefficients, {}, 'coefficients image');
    +

    If you click over a point using the Inspector tab, you will see the pixel values for the array image (coefficients “t” and “constant”, and residuals) and two-band image (coefficients “t” and “constant”) (Fig. F4.6.4).

    +
    +
    +

    +

    Fig. F4.6.4 Pixel values of array image and coefficients image

    +
    +
    +

    Now, copy and paste the code below to use the model to detrend the original NDVI time series and plot the time series chart with the trendlines parameter (Fig. F4.6.5).

    +
    // Compute a detrended series.  
+var detrended = landsat8sr.map(function(image) {   return image.select(dependent).subtract(  
+           image.select(independents).multiply(coefficients)  
+           .reduce('sum'))  
+       .rename(dependent)  
+       .copyProperties(image, ['system:time_start']);  
+});  
+  
+// Plot the detrended results.  
+var detrendedChart = ui.Chart.image.series(detrended, roi, null, 30)  
+   .setOptions({  
+       title: 'Detrended Landsat time series at ROI',  
+       lineWidth: 1,  
+       pointSize: 3,  
+       trendlines: {           0: {  
+               color: 'CC0000'           }  
+       },  
+   });print(detrendedChart);
    +
    +
    +

    +

    Fig. F4.6.5 Detrended NDVI time series

    +
    +
    @@ -1792,71 +2130,92 @@ Note
    -

    Code Checkpoint F46b. The book’s repository contains a script that shows what your code should look like at this point.

    +

    Code Checkpoint F46b. The book’s repository contains a script that shows what your code should look like at this point.

    12.4 Estimating Seasonality with a Harmonic Model

    -

    A linear trend is one of several possible types of trends in time series. Time series can also present harmonic trends, in which a value goes up and down in a predictable wave pattern. These are of particular interest and usefulness in the natural world, where harmonic changes in greenness of deciduous vegetation can occur across the spring, summer, and autumn. Now we will return to the initial time series (landsat8sr) of Fig. F4.6.2 and fit a harmonic pattern through the data. Consider the following harmonic model, where A is amplitude, ω is frequency, φ is phase, and et is a random error.

    -

    pt = β0 + β1t + Acos(2πωt - φ) + et 

    -

        = β0 + β1t + β2cos(2πωt) + β3sin(2πωt) + et                (Eq. F4.6.2)

    -

    Note that β2 = Acos(φ) and β3 = Asin(φ), implying A = (β22 + β32)½ and φ = atan(β3/β2) (as described in Shumway and Stoffer 2019). To fit this model to an annual time series, set ω = 1 (one cycle per year) and use ordinary least squares regression.

    -

    The setup for fitting the model is to first add the harmonic variables (the third and fourth terms of Eq. F4.6.2) to the ImageCollection. Then, fit the model as with the linear trend, using the linearRegression reducer, which will yield a 4 x 1 array image.

    -

    ///////////////////// Section 4 /////////////////////////////

    -

    // Use these independent variables in the harmonic regression.
    -var harmonicIndependents = ee.List([‘constant’, ‘t’, ‘cos’, ‘sin’]);

    -

    // Add harmonic terms as new image bands.
    -var harmonicLandsat = landsat8sr.map(function(image) {   var timeRadians = image.select(‘t’).multiply(2 * Math.PI);   return image       .addBands(timeRadians.cos().rename(‘cos’))
    -       .addBands(timeRadians.sin().rename(‘sin’));
    -});

    -

    // Fit the model.
    -var harmonicTrend = harmonicLandsat
    -   .select(harmonicIndependents.add(dependent))   // The output of this reducer is a 4x1 array image.   .reduce(ee.Reducer.linearRegression(harmonicIndependents.length(),       1));

    -

    Now, copy and paste the code below to plug the coefficients into Eq. F4.6.2 in order to get a time series of fitted values and plot the harmonic model time series (Fig. F4.6.6).

    -

    // Turn the array image into a multi-band image of coefficients.
    -var harmonicTrendCoefficients = harmonicTrend.select(‘coefficients’)
    -   .arrayProject([0])
    -   .arrayFlatten([harmonicIndependents]);

    -

    // Compute fitted values.
    -var fittedHarmonic = harmonicLandsat.map(function(image) {   return image.addBands(
    -       image.select(harmonicIndependents)
    -       .multiply(harmonicTrendCoefficients)
    -       .reduce(‘sum’)
    -       .rename(‘fitted’));
    -});

    -

    // Plot the fitted model and the original data at the ROI.
    -print(ui.Chart.image.series(
    -       fittedHarmonic.select([‘fitted’, ‘NDVI’]), roi, ee.Reducer
    -       .mean(), 30)
    -   .setSeriesNames([‘NDVI’, ‘fitted’])
    -   .setOptions({
    -       title: ‘Harmonic model: original and fitted values’,
    -       lineWidth: 1,
    -       pointSize: 3,
    -   }));

    -

    -

    Fig. F4.6.6 Harmonic model of NDVI time series

    -

    Returning to the mind-bending nature of curve-fitting, it is worth remembering that the harmonic waves seen in Fig. F4.6.6 are the fit of the data to a single point across the image. Next, we will map the outcomes of millions of these fits, pixel by pixel, across the entire study area.

    +

    A linear trend is one of several possible types of trends in time series. Time series can also present harmonic trends, in which a value goes up and down in a predictable wave pattern. These are of particular interest and usefulness in the natural world, where harmonic changes in greenness of deciduous vegetation can occur across the spring, summer, and autumn. Now we will return to the initial time series (landsat8sr) of Fig. F4.6.2 and fit a harmonic pattern through the data. Consider the following harmonic model, where A is amplitude, ω is frequency, φ is phase, and et is a random error.

    +

    pt = β0 + β1t + Acos(2πωt - φ) + et

    +
    = β0 + β1t + β2cos(2πωt) + β3sin(2πωt) + et                (Eq. F4.6.2)
    +

    Note that β2 = Acos(φ) and β3 = Asin(φ), implying A = (β22 + β32)½ and φ = atan(β3/β2) (as described in Shumway and Stoffer 2019). To fit this model to an annual time series, set ω = 1 (one cycle per year) and use ordinary least squares regression.

    +

    The setup for fitting the model is to first add the harmonic variables (the third and fourth terms of Eq. F4.6.2) to the ImageCollection. Then, fit the model as with the linear trend, using the linearRegression reducer, which will yield a 4 x 1 array image.

    +
    ///////////////////// Section 4 /////////////////////////////  
+  
+// Use these independent variables in the harmonic regression.  
+var harmonicIndependents = ee.List(['constant', 't', 'cos', 'sin']);  
+  
+// Add harmonic terms as new image bands.  
+var harmonicLandsat = landsat8sr.map(function(image) {   var timeRadians = image.select('t').multiply(2 * Math.PI);   return image       .addBands(timeRadians.cos().rename('cos'))  
+       .addBands(timeRadians.sin().rename('sin'));  
+});  
+  
+// Fit the model.  
+var harmonicTrend = harmonicLandsat  
+   .select(harmonicIndependents.add(dependent))   // The output of this reducer is a 4x1 array image.   .reduce(ee.Reducer.linearRegression(harmonicIndependents.length(),       1));
    +

    Now, copy and paste the code below to plug the coefficients into Eq. F4.6.2 in order to get a time series of fitted values and plot the harmonic model time series (Fig. F4.6.6).

    +
    // Turn the array image into a multi-band image of coefficients.  
+var harmonicTrendCoefficients = harmonicTrend.select('coefficients')  
+   .arrayProject([0])  
+   .arrayFlatten([harmonicIndependents]);  
+  
+// Compute fitted values.  
+var fittedHarmonic = harmonicLandsat.map(function(image) {   return image.addBands(  
+       image.select(harmonicIndependents)  
+       .multiply(harmonicTrendCoefficients)  
+       .reduce('sum')  
+       .rename('fitted'));  
+});  
+  
+// Plot the fitted model and the original data at the ROI.  
+print(ui.Chart.image.series(  
+       fittedHarmonic.select(['fitted', 'NDVI']), roi, ee.Reducer  
+       .mean(), 30)  
+   .setSeriesNames(['NDVI', 'fitted'])  
+   .setOptions({  
+       title: 'Harmonic model: original and fitted values',  
+       lineWidth: 1,  
+       pointSize: 3,  
+   }));
    +
    +
    +

    +

    Fig. F4.6.6 Harmonic model of NDVI time series

    +
    +
    +

    Returning to the mind-bending nature of curve-fitting, it is worth remembering that the harmonic waves seen in Fig. F4.6.6 are the fit of the data to a single point across the image. Next, we will map the outcomes of millions of these fits, pixel by pixel, across the entire study area.

    We’ll compute and map the phase and amplitude of the estimated harmonic model for each pixel. Phase and amplitude (Fig. F4.6.7) can give us additional information to facilitate remote sensing applications such as agricultural mapping and land use and land cover monitoring. Agricultural crops with different phenological cycles can be distinguished with phase and amplitude information, something that perhaps would not be possible with spectral information alone.

    -

    -

    Fig. F4.6.7 Example of phase and amplitude in harmonic model

    +
    +
    +

    +

    Fig. F4.6.7 Example of phase and amplitude in harmonic model

    +
    +

    Copy and paste the code below to compute phase and amplitude from the coefficients and add this image to the map (Fig. F4.6.8).

    -

    // Compute phase and amplitude.
    -var phase = harmonicTrendCoefficients.select(‘sin’)
    -   .atan2(harmonicTrendCoefficients.select(‘cos’))   // Scale to [0, 1] from radians.   .unitScale(-Math.PI, Math.PI);

    -

    var amplitude = harmonicTrendCoefficients.select(‘sin’)
    -   .hypot(harmonicTrendCoefficients.select(‘cos’))   // Add a scale factor for visualization.   .multiply(5);

    -

    // Compute the mean NDVI.
    -var meanNdvi = landsat8sr.select(‘NDVI’).mean();

    -

    // Use the HSV to RGB transformation to display phase and amplitude.
    -var rgb = ee.Image.cat([
    -   phase, // hue   amplitude, // saturation (difference from white)   meanNdvi // value (difference from black)
    -]).hsvToRgb();

    -

    Map.addLayer(rgb, {}, ‘phase (hue), amplitude (sat), ndvi (val)’);

    -

    -

    Fig. F4.6.8 Phase, amplitude, and NDVI concatenated image

    -

    The code uses the HSV to RGB transformation hsvToRgb for visualization purposes (Chap. F3.1). We use this transformation to separate color components from intensity for a better visualization. Without this transformation, we would visualize a very colorful image that would not look as intuitive as the image with the transformation. With this transformation, phase, amplitude, and mean NDVI are displayed in terms of hue (color), saturation (difference from white), and value (difference from black), respectively. Therefore, darker pixels are areas with low NDVI. For example, water bodies will appear as black, since NDVI values are zero or negative. The different colors are distinct phase values, and the saturation of the color refers to the amplitude: whiter colors mean amplitude closer to zero (e.g., forested areas), and the more vivid the colors, the higher the amplitude (e.g., croplands). Note that if you use the Inspector tool to analyze the values of a pixel, you will not get values of phase, amplitude, and NDVI, but the transformed values into values of blue, green, and red colors.

    +
    // Compute phase and amplitude.  
+var phase = harmonicTrendCoefficients.select('sin')  
+   .atan2(harmonicTrendCoefficients.select('cos'))   // Scale to [0, 1] from radians.   .unitScale(-Math.PI, Math.PI);  
+  
+var amplitude = harmonicTrendCoefficients.select('sin')  
+   .hypot(harmonicTrendCoefficients.select('cos'))   // Add a scale factor for visualization.   .multiply(5);  
+  
+// Compute the mean NDVI.  
+var meanNdvi = landsat8sr.select('NDVI').mean();  
+  
+// Use the HSV to RGB transformation to display phase and amplitude.  
+var rgb = ee.Image.cat([  
+   phase, // hue   amplitude, // saturation (difference from white)   meanNdvi // value (difference from black)  
+]).hsvToRgb();  
+  
+Map.addLayer(rgb, {}, 'phase (hue), amplitude (sat), ndvi (val)');
    +
    +
    +

    +

    Fig. F4.6.8 Phase, amplitude, and NDVI concatenated image

    +
    +
    +

    The code uses the HSV to RGB transformation hsvToRgb for visualization purposes (Chap. F3.1). We use this transformation to separate color components from intensity for a better visualization. Without this transformation, we would visualize a very colorful image that would not look as intuitive as the image with the transformation. With this transformation, phase, amplitude, and mean NDVI are displayed in terms of hue (color), saturation (difference from white), and value (difference from black), respectively. Therefore, darker pixels are areas with low NDVI. For example, water bodies will appear as black, since NDVI values are zero or negative. The different colors are distinct phase values, and the saturation of the color refers to the amplitude: whiter colors mean amplitude closer to zero (e.g., forested areas), and the more vivid the colors, the higher the amplitude (e.g., croplands). Note that if you use the Inspector tool to analyze the values of a pixel, you will not get values of phase, amplitude, and NDVI, but the transformed values into values of blue, green, and red colors.

    @@ -1867,7 +2226,7 @@ Note
    -

    Code Checkpoint F46c. The book’s repository contains a script that shows what your code should look like at this point.

    +

    Code Checkpoint F46c. The book’s repository contains a script that shows what your code should look like at this point.

    @@ -1875,24 +2234,33 @@ Note

    12.5 An Application of Curve Fitting

    12.5.1 The rich data about the curve fits can be viewed in a multitude of different ways. Add the code below to your script to produce the view in Fig. 4.6.9. The image will be a close-up of the area around Modesto, California.

    -

    ///////////////////// Section 5 /////////////////////////////

    -

    // Import point of interest over California, USA.
    -var roi = ee.Geometry.Point([-121.04, 37.641]);

    -

    // Set map center over the ROI.
    -Map.centerObject(roi, 14);

    -

    var trend0D = trend.select(‘coefficients’).arrayProject([0])
    -   .arrayFlatten([independents]).select(‘t’);

    -

    var anotherView = ee.Image(harmonicTrendCoefficients.select(‘sin’))
    -   .addBands(trend0D)
    -   .addBands(harmonicTrendCoefficients.select(‘cos’));

    -

    Map.addLayer(anotherView,
    -   {
    -       min: -0.03,
    -       max: 0.03   },   ‘Another combination of fit characteristics’);

    +
    ///////////////////// Section 5 /////////////////////////////  
+  
+// Import point of interest over California, USA.  
+var roi = ee.Geometry.Point([-121.04, 37.641]);  
+  
+// Set map center over the ROI.  
+Map.centerObject(roi, 14);  
+  
+var trend0D = trend.select('coefficients').arrayProject([0])  
+   .arrayFlatten([independents]).select('t');  
+  
+var anotherView = ee.Image(harmonicTrendCoefficients.select('sin'))  
+   .addBands(trend0D)  
+   .addBands(harmonicTrendCoefficients.select('cos'));  
+  
+Map.addLayer(anotherView,  
+   {  
+       min: -0.03,  
+       max: 0.03   },   'Another combination of fit characteristics');

    -

    -

    Fig. F4.6.9 Two views of the harmonic fits for NDVI for the Modesto, California area

    -

    The upper image in Fig. F4.6.9 is a closer view of Fig. F4.6.8, showing an image that transforms the sine and cosine coefficient values, and incorporates information from the mean NDVI. The lower image draws the sine and cosine in the red and blue bands, and extracts the slope of the linear trend that you calculated earlier in the chapter, placing that in the green band. The two views of the fit are similarly structured in their spatial pattern—both show fields to the west and the city to the east. But the pixel-by-pixel variability emphasizes a key point of this chapter: that a fit to the NDVI data is done independently in each pixel in the image. Using different elements of the fit, these two views, like other combinations of the data you might imagine, can reveal the rich variability of the landscape around Modesto.

    +
    +
    +

    +

    Fig. F4.6.9 Two views of the harmonic fits for NDVI for the Modesto, California area

    +
    +
    +

    The upper image in Fig. F4.6.9 is a closer view of Fig. F4.6.8, showing an image that transforms the sine and cosine coefficient values, and incorporates information from the mean NDVI. The lower image draws the sine and cosine in the red and blue bands, and extracts the slope of the linear trend that you calculated earlier in the chapter, placing that in the green band. The two views of the fit are similarly structured in their spatial pattern—both show fields to the west and the city to the east. But the pixel-by-pixel variability emphasizes a key point of this chapter: that a fit to the NDVI data is done independently in each pixel in the image. Using different elements of the fit, these two views, like other combinations of the data you might imagine, can reveal the rich variability of the landscape around Modesto.

    @@ -1903,13 +2271,13 @@ Note
    -

    Code Checkpoint F46d. The book’s repository contains a script that shows what your code should look like at this point.

    +

    Code Checkpoint F46d. The book’s repository contains a script that shows what your code should look like at this point.

    -

    12.6 Higher-Order Harmonic Models

    +

    12.6 Higher-Order Harmonic Models

    Harmonic models are not limited to fitting a single wave through a set of points. In some situations, there may be more than one cycle within a given year—for example, when an agricultural field is double-cropped. Modeling multiple waves within a given year can be done by adding more harmonic terms to Eq. F4.6.2. The code at the following checkpoint allows the fitting of any number of cycles through a given point.

    @@ -1921,18 +2289,22 @@ Note
    -

    Code Checkpoint F46e. The book’s repository contains a script to use to begin this section. You will need to start with that script and edit the code to produce the charts in this section.

    +

    Code Checkpoint F46e. The book’s repository contains a script to use to begin this section. You will need to start with that script and edit the code to produce the charts in this section.

    -

    Beginning with the repository script, changing the value of the harmonics variable will change the complexity of the harmonic curve fit by superimposing more or fewer harmonic waves on each other. While fitting higher-order functions improves the goodness-of-fit of the model to a given set of data, many of the coefficients may be close to zero at higher numbers or harmonic terms. Fig. F4.6.10 shows the fit through the example point using one, two, and three harmonic curves.

    +

    Beginning with the repository script, changing the value of the harmonics variable will change the complexity of the harmonic curve fit by superimposing more or fewer harmonic waves on each other. While fitting higher-order functions improves the goodness-of-fit of the model to a given set of data, many of the coefficients may be close to zero at higher numbers or harmonic terms. Fig. F4.6.10 shows the fit through the example point using one, two, and three harmonic curves.

    -

    -

    Fig. F4.6.10 Fit with harmonic curves of increasing complexity, fitted for data at a given point 

    +
    +
    +

    +

    Fig. F4.6.10 Fit with harmonic curves of increasing complexity, fitted for data at a given point

    +
    +

    Synthesis

    -

    Assignment 1. Fit two NDVI harmonic models for a point close to Manaus, Brazil: one prior to a disturbance event and one after the disturbance event (Fig. F4.6.11). You can start with the code checkpoint below, which gives you the point coordinates and defines the initial functions needed. The disturbance event happened in mid-December 2014, so set filter dates for the first ImageCollection to ‘2013-01-01’,‘2014-12-12’, and set the filter dates for the second ImageCollection to ‘2014-12-13’,‘2019-01-01’. Merge both fitted collections and plot both NDVI and fitted values. The result should look like Fig. F4.6.12.

    +

    Assignment 1. Fit two NDVI harmonic models for a point close to Manaus, Brazil: one prior to a disturbance event and one after the disturbance event (Fig. F4.6.11). You can start with the code checkpoint below, which gives you the point coordinates and defines the initial functions needed. The disturbance event happened in mid-December 2014, so set filter dates for the first ImageCollection to ‘2013-01-01’,‘2014-12-12’, and set the filter dates for the second ImageCollection to ‘2014-12-13’,‘2019-01-01’. Merge both fitted collections and plot both NDVI and fitted values. The result should look like Fig. F4.6.12.

    @@ -1943,14 +2315,22 @@ Note
    -

    Code Checkpoint F46s1. The book’s repository contains a script that shows what your code should look like at this point.

    +

    Code Checkpoint F46s1. The book’s repository contains a script that shows what your code should look like at this point.

    -

    -

    Fig. F4.6.11 Landsat 8 images showing the land cover change at a point in Manaus, Brazil; (left) July, 6, 2014, (right) August 8, 2015

    -

    -

    Fig. F4.6.12 Fitted harmonic models before and after disturbance events to a given point in the Brazilian Amazon

    -

    What do you notice? Think about how the harmonic model would look if you tried to fit the entire period. In this example, you were given the date of the breakpoint between the two conditions of the land surface within the time series. State-of-the-art land cover change algorithms work by assessing the difference between the modeled and observed pixel values. These algorithms look for breakpoints in the model, typically flagging changes after a predefined number of consecutive observations.

    +
    +
    +

    +

    Fig. F4.6.11 Landsat 8 images showing the land cover change at a point in Manaus, Brazil; (left) July, 6, 2014, (right) August 8, 2015

    +
    +
    +
    +
    +

    +

    Fig. F4.6.12 Fitted harmonic models before and after disturbance events to a given point in the Brazilian Amazon

    +
    +
    +

    What do you notice? Think about how the harmonic model would look if you tried to fit the entire period. In this example, you were given the date of the breakpoint between the two conditions of the land surface within the time series. State-of-the-art land cover change algorithms work by assessing the difference between the modeled and observed pixel values. These algorithms look for breakpoints in the model, typically flagging changes after a predefined number of consecutive observations.

    @@ -1961,13 +2341,13 @@ Note
    -

    Code Checkpoint F46s2. The book’s repository contains a script that shows what your code should look like at this point.

    +

    Code Checkpoint F46s2. The book’s repository contains a script that shows what your code should look like at this point.

    Conclusion

    -

    In this chapter, we learned how to graph and fit both linear and harmonic functions to time series of remotely sensed data. These skills underpin important tools such as Continuous Change Detection and Classification (CCDC, Chap. F4.7) and Continuous Degradation Detection (CODED, Chap. A3.4). These approaches are used by many organizations to detect forest degradation and deforestation (e.g., Tang et al. 2019, Bullock et al. 2020). These approaches can also be used to identify crops (Chap. A1.1) with high degrees of accuracy (Ghazaryan et al. 2018).

    +

    In this chapter, we learned how to graph and fit both linear and harmonic functions to time series of remotely sensed data. These skills underpin important tools such as Continuous Change Detection and Classification (CCDC, Chap. F4.7) and Continuous Degradation Detection (CODED, Chap. A3.4). These approaches are used by many organizations to detect forest degradation and deforestation (e.g., Tang et al. 2019, Bullock et al. 2020). These approaches can also be used to identify crops (Chap. A1.1) with high degrees of accuracy (Ghazaryan et al. 2018).

    References

    @@ -1980,10 +2360,18 @@ Note

    13 Interpreting Time Series with CCDC

    -

    ::: {.callout-tip} # Chapter Information

    +
    +
    +
    + +
    +
    +Chapter Information +
    +
    +

    Author

    -

     

    Paulo Arévalo, Pontus Olofsson

    @@ -2013,7 +2401,7 @@ Note
    -

    13.1 Introduction to Theory 

    +

    13.1 Introduction to Theory

    “A time series is a sequence of observations taken sequentially in time. … An intrinsic feature of a time series is that, typically, adjacent observations are dependent. Time-series analysis is concerned with techniques for the analysis of this dependency.” This is the formal definition of time-series analysis by Box et al. (1994). In a remote sensing context, the observations of interest are measurements of radiation reflected from the surface of the Earth from the Sun or an instrument emitting energy toward Earth. Consecutive measurements made over a given area result in a time series of surface reflectance. By analyzing such time series, we can achieve a comprehensive characterization of ecosystem and land surface processes (Kennedy et al. 2014). The result is a shift away from traditional, retrospective change-detection approaches based on data acquired over the same area at two or a few points in time to continuous monitoring of the landscape (Woodcock et al. 2020). Previous obstacles related to data storage, preprocessing, and computing power have been largely overcome with the emergence of powerful cloud-computing platforms that provide direct access to the data (Gorelick et al. 2017). In this chapter, we will illustrate how to study landscape dynamics in the Amazon river basin by analyzing dense time series of Landsat data using the CCDC algorithm. Unlike LandTrendr (Chap. F4.5), which uses anniversary images to fit straight line segments that describe the spectral trajectory over time, CCDC uses all available clear observations. This has multiple advantages, including the ability to detect changes within a year and capture seasonal patterns, although at the expense of much higher computational demands and more complexity to manipulate the outputs, compared to LandTrendr.

    @@ -2029,67 +2417,74 @@ Note
    -

    Code Checkpoint F47a. The book’s repository contains information about accessing the CCDC interface.

    +

    Code Checkpoint F47a. The book’s repository contains information about accessing the CCDC interface.

    -

    Once you have loaded the CCDC interface (Fig. F4.7.1), you will be able to navigate to any location, pick a Landsat spectral band or index to plot, and click on the map to see the fit by CCDC at the location you clicked. For this exercise, we will study landscape dynamics in the state of Rondônia, Brazil. We can use the panel on the left-bottom corner to enter the following coordinates (latitude, longitude): -9.0002, -62.7223. A point will be added in that location and the map will zoom in to it. Once there, click on the point and wait for the chart at the bottom to load. This example shows the Landsat time series for the first shortwave infrared (SWIR1) band (as blue dots) and the time segments (as colored lines) run using CCDC default parameters. The first segment represents stable forest, which was abruptly cut in mid-2006. The algorithm detects this change event and fits a new segment afterwards, representing a new temporal pattern of agriculture. Other subsequent patterns are detected as new segments are fitted that may correspond to cycles of harvest and regrowth, or a different crop. To investigate the dynamics over time, you can click on the points in the chart, and the Landsat images they correspond to will be added to the map according to the visualization parameters selected for the RGB combination in the left panel. Currently, changes made in that panel are not immediate but must be set before clicking on the map. 

    +

    Once you have loaded the CCDC interface (Fig. F4.7.1), you will be able to navigate to any location, pick a Landsat spectral band or index to plot, and click on the map to see the fit by CCDC at the location you clicked. For this exercise, we will study landscape dynamics in the state of Rondônia, Brazil. We can use the panel on the left-bottom corner to enter the following coordinates (latitude, longitude): -9.0002, -62.7223. A point will be added in that location and the map will zoom in to it. Once there, click on the point and wait for the chart at the bottom to load. This example shows the Landsat time series for the first shortwave infrared (SWIR1) band (as blue dots) and the time segments (as colored lines) run using CCDC default parameters. The first segment represents stable forest, which was abruptly cut in mid-2006. The algorithm detects this change event and fits a new segment afterwards, representing a new temporal pattern of agriculture. Other subsequent patterns are detected as new segments are fitted that may correspond to cycles of harvest and regrowth, or a different crop. To investigate the dynamics over time, you can click on the points in the chart, and the Landsat images they correspond to will be added to the map according to the visualization parameters selected for the RGB combination in the left panel. Currently, changes made in that panel are not immediate but must be set before clicking on the map.

    Pay special attention to the characteristics of each segment. For example, look at the average surface reflectance value for each segment. The presence of a pronounced slope may be indicative of phenomena like vegetation regrowth or degradation. The number of harmonics used in each segment may represent seasonality in vegetation (either natural or due to agricultural practices) or landscape dynamics (e.g., seasonal flooding).

    -

    -

    Fig. 4.7.1 Landsat time series for the SWIR1 band (blue dots) and CCDC time segments (colored lines) showing a forest loss event circa 2006 for a place in Rondônia, Brazil

    -

    Question 1. While still using the SWIR1 band, click on a pixel that is forested. What do the time series and time segments look like?

    +
    +
    +

    +

    Fig. 4.7.1 Landsat time series for the SWIR1 band (blue dots) and CCDC time segments (colored lines) showing a forest loss event circa 2006 for a place in Rondônia, Brazil

    +
    +
    +

    Question 1. While still using the SWIR1 band, click on a pixel that is forested. What do the time series and time segments look like?

    13.3 Running CCDC

    -

    The tool shown above is useful for understanding the temporal dynamics for a specific point. However, we can do a similar analysis for larger areas by first running the CCDC algorithm over a group of pixels. The CCDC function in Earth Engine can take any ImageCollection, ideally one with little or no noise, such as a Landsat ImageCollection where clouds and cloud shadows have been masked. CCDC contains an internal cloud masking algorithm and is rather robust against missed clouds, but the cleaner the data the better. To simplify the process, we have developed a function library that contains functions for generating input data and processing CCDC results. Paste this line of code in a new script:

    -

    var utils = require(   ‘users/parevalo_bu/gee-ccdc-tools:ccdcUtilities/api’);

    -

    For the current exercise, we will obtain an ImageCollection of Landsat 4, 5, 7, and 8 data (Collection 2 Tier 1) that has been filtered for clouds, cloud shadows, haze, and radiometrically saturated pixels. If we were to do this manually, we would retrieve each ImageCollection for each satellite, apply the corresponding filters and then merge them all into a single ImageCollection. Instead, to simplify that process, we will use the function getLandsat, included in the “Inputs” module of our utilities, and then filter the resulting ImageCollection to a small study region for the period between 2000 and 2020. The getLandsat function will retrieve all surface reflectance bands (renamed and scaled to actual surface reflectance units) as well as other vegetation indices. To simplify the exercise, we will select only the surface reflectance bands we are going to use, adding the following code to your script:

    -

    var studyRegion = ee.Geometry.Rectangle([
    -   [-63.9533, -10.1315],
    -   [-64.9118, -10.6813]
    +

    The tool shown above is useful for understanding the temporal dynamics for a specific point. However, we can do a similar analysis for larger areas by first running the CCDC algorithm over a group of pixels. The CCDC function in Earth Engine can take any ImageCollection, ideally one with little or no noise, such as a Landsat ImageCollection where clouds and cloud shadows have been masked. CCDC contains an internal cloud masking algorithm and is rather robust against missed clouds, but the cleaner the data the better. To simplify the process, we have developed a function library that contains functions for generating input data and processing CCDC results. Paste this line of code in a new script:

    +

    var utils = require( ‘users/parevalo_bu/gee-ccdc-tools:ccdcUtilities/api’);

    +

    For the current exercise, we will obtain an ImageCollection of Landsat 4, 5, 7, and 8 data (Collection 2 Tier 1) that has been filtered for clouds, cloud shadows, haze, and radiometrically saturated pixels. If we were to do this manually, we would retrieve each ImageCollection for each satellite, apply the corresponding filters and then merge them all into a single ImageCollection. Instead, to simplify that process, we will use the function getLandsat, included in the “Inputs” module of our utilities, and then filter the resulting ImageCollection to a small study region for the period between 2000 and 2020. The getLandsat function will retrieve all surface reflectance bands (renamed and scaled to actual surface reflectance units) as well as other vegetation indices. To simplify the exercise, we will select only the surface reflectance bands we are going to use, adding the following code to your script:

    +

    var studyRegion = ee.Geometry.Rectangle([
    +[-63.9533, -10.1315],
    +[-64.9118, -10.6813]
    ]);

    -

    // Define start, end dates and Landsat bands to use.
    -var startDate = ‘2000-01-01’;
    -var endDate = ‘2020-01-01’;
    -var bands = [‘BLUE’, ‘GREEN’, ‘RED’, ‘NIR’, ‘SWIR1’, ‘SWIR2’];

    -

    // Retrieve all clear, Landsat 4, 5, 7 and 8 observations (Collection 2, Tier 1).
    -var filteredLandsat = utils.Inputs.getLandsat({
    -       collection: 2   })
    -   .filterBounds(studyRegion)
    -   .filterDate(startDate, endDate)
    -   .select(bands);

    -

    print(filteredLandsat.first());

    -

    With the ImageCollection ready, we can specify the CCDC parameters and run the algorithm. For this exercise we will use the default parameters, which tend to work reasonably well in most circumstances. The only parameters we will modify are the breakpoint bands, date format, and lambda. We will set all the parameter values in a dictionary that we will pass to the CCDC function. For the break detection process we use all bands except for the blue and surface temperature bands (‘BLUE’ and ‘TEMP’, respectively). The minObservations default value of 6 represents the number of consecutive observations required to flag a change. The chiSquareProbability and minNumOfYearsScaler default parameters of 0.99 and 1.33, respectively, control the sensitivity of the algorithm to detect change and the iterative curve fitting process required to detect change. We set the date format to 1, which corresponds to fractional years and tends to be easier to interpret. For instance, a change detected in the middle day of the year 2010 would be stored in a pixel as 2010.5. Finally, we use the default value of lambda of 20, but we scale it to match the scale of the inputs (surface reflectance units), and we specify a maxIterations value of 10000, instead of the default of 25000, which might take longer to complete. Those two parameters control the curve fitting process.

    -

    To complete the input parameters, we specify the ImageCollection to use, which we derived in the previous code section. Add this code below:

    -

    // Set CCD params to use.
    -var ccdParams = {
    -   breakpointBands: [‘GREEN’, ‘RED’, ‘NIR’, ‘SWIR1’, ‘SWIR2’],
    -   tmaskBands: [‘GREEN’, ‘SWIR2’],
    -   minObservations: 6,
    -   chiSquareProbability: 0.99,
    -   minNumOfYearsScaler: 1.33,
    -   dateFormat: 1,
    -   lambda: 0.002,
    -   maxIterations: 10000,
    -   collection: filteredLandsat
    -};

    -

    // Run CCD.
    -var ccdResults = ee.Algorithms.TemporalSegmentation.Ccdc(ccdParams);
    -print(ccdResults);

    -

    Notice that the output ccdResults contains a large number of bands, with some of them corresponding to two-dimensional arrays. We will explore these bands more in the following section. The process of running the algorithm interactively for more than a handful of pixels can become very taxing to the system very quickly, resulting in memory errors. To avoid having such issues, we typically export the results to an Earth Engine asset first, and then inspect the asset. This approach ensures that CCDC completes its run successfully, and also allows us to access the results easily later. In the following sections of this chapter, we will use a precomputed asset, instead of asking you to export the asset yourself. For your reference, the code required to export CCDC results is shown below, with the flag set to false to help you remember to not export the results now, but instead to use the precomputed asset in the following sections.

    -

    var exportResults = false
    -if (exportResults) {   // Create a metadata dictionary with the parameters and arguments used.   var metadata = ccdParams;
    -   metadata[‘breakpointBands’] =
    -       metadata[‘breakpointBands’].toString();
    -   metadata[‘tmaskBands’] = metadata[‘tmaskBands’].toString();
    -   metadata[‘startDate’] = startDate;
    -   metadata[‘endDate’] = endDate;
    -   metadata[‘bands’] = bands.toString();   // Export results, assigning the metadata as image properties.   //   Export.image.toAsset({
    -       image: ccdResults.set(metadata),
    -       region: studyRegion,
    -       pyramidingPolicy: {           “.default”: ‘sample’       },
    -       scale: 30   });
    +

    // Define start, end dates and Landsat bands to use.  
+var startDate = '2000-01-01';  
+var endDate = '2020-01-01';  
+var bands = ['BLUE', 'GREEN', 'RED', 'NIR', 'SWIR1', 'SWIR2'];  
+  
+// Retrieve all clear, Landsat 4, 5, 7 and 8 observations (Collection 2, Tier 1).  
+var filteredLandsat = utils.Inputs.getLandsat({  
+       collection: 2   })  
+   .filterBounds(studyRegion)  
+   .filterDate(startDate, endDate)  
+   .select(bands);  
+  
+print(filteredLandsat.first());
    +

    With the ImageCollection ready, we can specify the CCDC parameters and run the algorithm. For this exercise we will use the default parameters, which tend to work reasonably well in most circumstances. The only parameters we will modify are the breakpoint bands, date format, and lambda. We will set all the parameter values in a dictionary that we will pass to the CCDC function. For the break detection process we use all bands except for the blue and surface temperature bands (‘BLUE’ and ‘TEMP’, respectively). The minObservations default value of 6 represents the number of consecutive observations required to flag a change. The chiSquareProbability and minNumOfYearsScaler default parameters of 0.99 and 1.33, respectively, control the sensitivity of the algorithm to detect change and the iterative curve fitting process required to detect change. We set the date format to 1, which corresponds to fractional years and tends to be easier to interpret. For instance, a change detected in the middle day of the year 2010 would be stored in a pixel as 2010.5. Finally, we use the default value of lambda of 20, but we scale it to match the scale of the inputs (surface reflectance units), and we specify a maxIterations value of 10000, instead of the default of 25000, which might take longer to complete. Those two parameters control the curve fitting process.

    +

    To complete the input parameters, we specify the ImageCollection to use, which we derived in the previous code section. Add this code below:

    +
    // Set CCD params to use.  
+var ccdParams = {  
+   breakpointBands: ['GREEN', 'RED', 'NIR', 'SWIR1', 'SWIR2'],  
+   tmaskBands: ['GREEN', 'SWIR2'],  
+   minObservations: 6,  
+   chiSquareProbability: 0.99,  
+   minNumOfYearsScaler: 1.33,  
+   dateFormat: 1,  
+   lambda: 0.002,  
+   maxIterations: 10000,  
+   collection: filteredLandsat  
+};  
+  
+// Run CCD.  
+var ccdResults = ee.Algorithms.TemporalSegmentation.Ccdc(ccdParams);  
+print(ccdResults);
    +

    Notice that the output ccdResults contains a large number of bands, with some of them corresponding to two-dimensional arrays. We will explore these bands more in the following section. The process of running the algorithm interactively for more than a handful of pixels can become very taxing to the system very quickly, resulting in memory errors. To avoid having such issues, we typically export the results to an Earth Engine asset first, and then inspect the asset. This approach ensures that CCDC completes its run successfully, and also allows us to access the results easily later. In the following sections of this chapter, we will use a precomputed asset, instead of asking you to export the asset yourself. For your reference, the code required to export CCDC results is shown below, with the flag set to false to help you remember to not export the results now, but instead to use the precomputed asset in the following sections.

    +

    var exportResults = false
    +if (exportResults) { // Create a metadata dictionary with the parameters and arguments used. var metadata = ccdParams;
    +metadata[‘breakpointBands’] =
    +metadata[‘breakpointBands’].toString();
    +metadata[‘tmaskBands’] = metadata[‘tmaskBands’].toString();
    +metadata[‘startDate’] = startDate;
    +metadata[‘endDate’] = endDate;
    +metadata[‘bands’] = bands.toString(); // Export results, assigning the metadata as image properties. // Export.image.toAsset({
    +image: ccdResults.set(metadata),
    +region: studyRegion,
    +pyramidingPolicy: { “.default”: ‘sample’ },
    +scale: 30 });
    }

    -

    Note the metadata variable above. This is not strictly required for exporting the per-pixel CCDC results, but it allows us to keep a record of important properties of the run by attaching this information as metadata to the image. Additionally, some of the tools we have created to interact with CCDC outputs use this user-created metadata to facilitate using the asset. Note also that setting the value of pyramidingPolicy to ‘sample’ ensures that all the bands in the output have the proper policy.

    +

    Note the metadata variable above. This is not strictly required for exporting the per-pixel CCDC results, but it allows us to keep a record of important properties of the run by attaching this information as metadata to the image. Additionally, some of the tools we have created to interact with CCDC outputs use this user-created metadata to facilitate using the asset. Note also that setting the value of pyramidingPolicy to ‘sample’ ensures that all the bands in the output have the proper policy.

    As a general rule, try to use pre-existing CCDC results if possible, and if you want to try running it yourself outside of this lab exercise, start with very small areas. For instance, the study area in this exercise would take approximately 30 minutes on average to export, but larger tiles may take several hours to complete, depending on the number of images in the collection and the parameters used.

    @@ -2101,16 +2496,16 @@ Note
    -

    Code Checkpoint F47b. The book’s repository contains a script that shows what your code should look like at this point.

    +

    Code Checkpoint F47b. The book’s repository contains a script that shows what your code should look like at this point.

    13.4 Extracting Break Information

    We will now start exploring the pre-exported CCDC results mentioned in the previous section. We will make use of the third-party module palettes, described in detail in Chap. F6.0, that simplifies the use of palettes for visualization. Paste the following code in a new script:

    -

    var palettes = require(‘users/gena/packages:palettes’);

    -

    var resultsPath =   ‘projects/gee-book/assets/F4-7/Rondonia_example_small’;
    -var ccdResults = ee.Image(resultsPath);
    +

    var palettes = require(‘users/gena/packages:palettes’);

    +

    var resultsPath = ‘projects/gee-book/assets/F4-7/Rondonia_example_small’;
    +var ccdResults = ee.Image(resultsPath);
    Map.centerObject(ccdResults, 10);
    print(ccdResults);

    The first line calls a library that will facilitate visualizing the images. The second line contains the path to the precomputed results of the CCDC run shown in the previous section. The printed asset will contain the following bands:

    @@ -2119,81 +2514,96 @@ print(ccdResults);

  • tEnd: The end date of each time segment
  • tBreak: The time segment break date if a change is detected
  • numObs: The number of observations used in each time segment
    • -
  • changeProb: A numeric value representing the change probability for each of the bands used for change detection
    • +
  • changeProb: A numeric value representing the change probability for each of the bands used for change detection
  • *_coefs: The regression coefficients for each of the bands in the input image collection
  • *_rmse: The model root-mean-square error for each time segment and input band
  • *_magnitude: For time segments with detected changes, this represents the normalized residuals during the change period
    • -

    Notice that next to the band name and band type, there is also the number of dimensions (i.e., 1 dimension, 2 dimensions). This is an indication that we are dealing with an array image, which typically requires a specific set of functions for proper manipulation, some of which we will use in the next steps. We will start by looking at the change bands, which are one of the key outputs of the CCDC algorithm. We will select the band containing the information on the timing of break, and find the number of breaks for a given time range. In the same script, paste the code below:

    -

    // Select time of break and change probability array images.
    -var change = ccdResults.select(‘tBreak’);
    -var changeProb = ccdResults.select(‘changeProb’);

    -

    // Set the time range we want to use and get as mask of
    -// places that meet the condition.
    -var start = 2000;
    -var end = 2021;
    -var mask = change.gt(start).and(change.lte(end)).and(changeProb.eq(
    -1));
    -Map.addLayer(changeProb, {}, ‘change prob’);

    -

    // Obtain the number of breaks for the time range.
    -var numBreaks = mask.arrayReduce(ee.Reducer.sum(), [0]);
    -Map.addLayer(numBreaks, {
    -   min: 0,
    -   max: 5}, ‘Number of breaks’);

    -

    With this code, we define the time range that we want to use, and then we generate a mask that will indicate all the positions in the image array with breaks detected in that range that also meet the condition of having a change probability of 1, effectively removing some spurious breaks. For each pixel, we can count the number of times that the mask retrieved a valid result, indicating the number of breaks detected by CCDC. In the loaded layer, places that appear brighter will show a higher number of breaks, potentially indicating the conversion from forest to agriculture, followed by multiple agricultural cycles. Keep in mind that the detection of a break does not always imply a change of land cover. Natural events, small-scale disturbances and seasonal cycles, among others, can result in the detection of a break by CCDC. Similarly, changes in the condition of the land cover in a pixel can also be detected as breaks by CCDC, and some erroneous breaks can also happen due to noisy time series or other factors.

    +

    Notice that next to the band name and band type, there is also the number of dimensions (i.e., 1 dimension, 2 dimensions). This is an indication that we are dealing with an array image, which typically requires a specific set of functions for proper manipulation, some of which we will use in the next steps. We will start by looking at the change bands, which are one of the key outputs of the CCDC algorithm. We will select the band containing the information on the timing of break, and find the number of breaks for a given time range. In the same script, paste the code below:

    +
    // Select time of break and change probability array images.  
+var change = ccdResults.select('tBreak');  
+var changeProb = ccdResults.select('changeProb');  
+  
+// Set the time range we want to use and get as mask of  
+// places that meet the condition.  
+var start = 2000;  
+var end = 2021;  
+var mask = change.gt(start).and(change.lte(end)).and(changeProb.eq(  
+1));  
+Map.addLayer(changeProb, {}, 'change prob');  
+  
+// Obtain the number of breaks for the time range.  
+var numBreaks = mask.arrayReduce(ee.Reducer.sum(), [0]);  
+Map.addLayer(numBreaks, {  
+   min: 0,  
+   max: 5}, 'Number of breaks');
    +

    With this code, we define the time range that we want to use, and then we generate a mask that will indicate all the positions in the image array with breaks detected in that range that also meet the condition of having a change probability of 1, effectively removing some spurious breaks. For each pixel, we can count the number of times that the mask retrieved a valid result, indicating the number of breaks detected by CCDC. In the loaded layer, places that appear brighter will show a higher number of breaks, potentially indicating the conversion from forest to agriculture, followed by multiple agricultural cycles. Keep in mind that the detection of a break does not always imply a change of land cover. Natural events, small-scale disturbances and seasonal cycles, among others, can result in the detection of a break by CCDC. Similarly, changes in the condition of the land cover in a pixel can also be detected as breaks by CCDC, and some erroneous breaks can also happen due to noisy time series or other factors.

    For places with many changes, visualizing the first or last time when a break was recorded can be helpful to understand the change dynamics happening in the landscape. Paste the code below in the same script:

    -

    // Obtain the first change in that time period.
    -var dates = change.arrayMask(mask).arrayPad([1]);
    -var firstChange = dates
    -   .arraySlice(0, 0, 1)
    -   .arrayFlatten([
    -       [‘firstChange’]
    -   ])
    -   .selfMask();

    -

    var timeVisParams = {
    -   palette: palettes.colorbrewer.YlOrRd[9],
    -   min: start,
    -   max: end
    -};
    -Map.addLayer(firstChange, timeVisParams, ‘First change’);

    -

    // Obtain the last change in that time period.
    -var lastChange = dates
    -   .arraySlice(0, -1)
    -   .arrayFlatten([
    -       [‘lastChange’]
    -   ])
    -   .selfMask();
    -Map.addLayer(lastChange, timeVisParams, ‘Last change’);

    -

    Here we use arrayMask to keep only the change dates that meet our condition, by using the mask we created previously. We use the function arrayPad to fill or “pad” those pixels that did not experience any change and therefore have no value in the tBreak band. Then we select either the first or last values in the array, and we convert the image from a one-dimensional array to a regular image, in order to apply a visualization to it, using a custom palette. The results should look like Fig. F4.7.2.

    +
    // Obtain the first change in that time period.  
+var dates = change.arrayMask(mask).arrayPad([1]);  
+var firstChange = dates  
+   .arraySlice(0, 0, 1)  
+   .arrayFlatten([  
+       ['firstChange']  
+   ])  
+   .selfMask();  
+  
+var timeVisParams = {  
+   palette: palettes.colorbrewer.YlOrRd[9],  
+   min: start,  
+   max: end  
+};  
+Map.addLayer(firstChange, timeVisParams, 'First change');  
+  
+// Obtain the last change in that time period.  
+var lastChange = dates  
+   .arraySlice(0, -1)  
+   .arrayFlatten([  
+       ['lastChange']  
+   ])  
+   .selfMask();  
+Map.addLayer(lastChange, timeVisParams, 'Last change');
    +

    Here we use arrayMask to keep only the change dates that meet our condition, by using the mask we created previously. We use the function arrayPad to fill or “pad” those pixels that did not experience any change and therefore have no value in the tBreak band. Then we select either the first or last values in the array, and we convert the image from a one-dimensional array to a regular image, in order to apply a visualization to it, using a custom palette. The results should look like Fig. F4.7.2.

    Finally, we can use the magnitude bands to visualize where and when the largest changes as recorded by CCDC have occurred, during our selected time period. We are going to use the magnitude of change in the SWIR1 band, masking it and padding it in the same way we did before. Paste this code in your script:

    -

    // Get masked magnitudes.
    -var magnitudes = ccdResults
    -   .select(‘SWIR1_magnitude’)
    -   .arrayMask(mask)
    -   .arrayPad([1]);

    -

    // Get index of max abs magnitude of change.
    -var maxIndex = magnitudes
    -   .abs()
    -   .arrayArgmax()
    -   .arrayFlatten([
    -       [‘index’]
    -   ]);

    -

    // Select max magnitude and its timing
    -var selectedMag = magnitudes.arrayGet(maxIndex);
    -var selectedTbreak = dates.arrayGet(maxIndex).selfMask();

    -

    var magVisParams = {
    -   palette: palettes.matplotlib.viridis[7],
    -   min: -0.15,
    -   max: 0.15
    -};
    -Map.addLayer(selectedMag, magVisParams, ‘Max mag’);
    -Map.addLayer(selectedTbreak, timeVisParams, ‘Time of max mag’);

    +
    // Get masked magnitudes.  
+var magnitudes = ccdResults  
+   .select('SWIR1_magnitude')  
+   .arrayMask(mask)  
+   .arrayPad([1]);  
+  
+// Get index of max abs magnitude of change.  
+var maxIndex = magnitudes  
+   .abs()  
+   .arrayArgmax()  
+   .arrayFlatten([  
+       ['index']  
+   ]);  
+  
+// Select max magnitude and its timing  
+var selectedMag = magnitudes.arrayGet(maxIndex);  
+var selectedTbreak = dates.arrayGet(maxIndex).selfMask();  
+  
+var magVisParams = {  
+   palette: palettes.matplotlib.viridis[7],  
+   min: -0.15,  
+   max: 0.15  
+};  
+Map.addLayer(selectedMag, magVisParams, 'Max mag');  
+Map.addLayer(selectedTbreak, timeVisParams, 'Time of max mag');

    -

    -

    Fig. F4.7.2 First (top) and last (bottom) detected breaks for the study area. Darker colors represent more recent dates, while brighter colors represent older dates. The first change layer shows the clear patterns of original agricultural expansion closer to the year 2000. The last change layer shows the more recently detected and noisy breaks in the same areas. The thin areas in the center of the image have only one time of change, corresponding to a single deforestation event. Pixels with no detected breaks are masked and therefore show the basemap underneath, set to show satellite imagery.

    +
    +
    +

    +

    Fig. F4.7.2 First (top) and last (bottom) detected breaks for the study area. Darker colors represent more recent dates, while brighter colors represent older dates. The first change layer shows the clear patterns of original agricultural expansion closer to the year 2000. The last change layer shows the more recently detected and noisy breaks in the same areas. The thin areas in the center of the image have only one time of change, corresponding to a single deforestation event. Pixels with no detected breaks are masked and therefore show the basemap underneath, set to show satellite imagery.

    +
    +

    We first take the absolute value because the magnitudes can be positive or negative, depending on the direction of the change and the band used. For example, a positive value in the SWIR1 may show a forest loss event, where surface reflectance goes from low to higher values. Brighter values in Fig. 4.7.3 represent events of that type. Conversely, a flooding event would have a negative value, due to the corresponding drop in reflectance. Once we find the maximum absolute value, we find its position on the array and then use that index to extract the original magnitude value, as well as the time when that break occurred.

    -

    -

    Fig. F4.7.3 Maximum magnitude of change for the SWIR1 band for the selected study period

    +
    +
    +

    +

    Fig. F4.7.3 Maximum magnitude of change for the SWIR1 band for the selected study period

    +
    +
    @@ -2204,59 +2614,64 @@ Note
    -

    Code Checkpoint F47c. The book’s repository contains a script that shows what your code should look like at this point.

    +

    Code Checkpoint F47c. The book’s repository contains a script that shows what your code should look like at this point.

    -

    Question 2. Compare the “first change” and “last change” layers with the layer showing the timing of the maximum magnitude of change. Use the Inspector to check the values for specific pixels if necessary. What does the timing of the layers tell you about the change processes happening in the area?

    +

    Question 2. Compare the “first change” and “last change” layers with the layer showing the timing of the maximum magnitude of change. Use the Inspector to check the values for specific pixels if necessary. What does the timing of the layers tell you about the change processes happening in the area?

    Question 3. Looking at the “max magnitude of change” layer, find places showing the largest and the smallest values. What type of changes do you think are happening in each of those places?

    13.5 Extracting Coefficients Manually

    -

    In addition to the change information generated by the CCDC algorithm, we can use the coefficients of the time segments for multiple purposes, like land cover classification. Each time segment can be described as a harmonic function with an intercept, slope, and three pairs of sine and cosine terms that allow the time segments to represent seasonality occurring at different temporal scales. These coefficients, as well as the root-mean-square error (RMSE) obtained by comparing each predicted and actual Landsat value, are produced when the CCDC algorithm is run. The following example will show you how to retrieve the intercept coefficient for a segment intersecting a specific date. In a new script, paste the code below:

    -

    var palettes = require(‘users/gena/packages:palettes’);

    -

    var resultsPath =   ‘projects/gee-book/assets/F4-7/Rondonia_example_small’;
    -var ccdResults = ee.Image(resultsPath);
    +

    In addition to the change information generated by the CCDC algorithm, we can use the coefficients of the time segments for multiple purposes, like land cover classification. Each time segment can be described as a harmonic function with an intercept, slope, and three pairs of sine and cosine terms that allow the time segments to represent seasonality occurring at different temporal scales. These coefficients, as well as the root-mean-square error (RMSE) obtained by comparing each predicted and actual Landsat value, are produced when the CCDC algorithm is run. The following example will show you how to retrieve the intercept coefficient for a segment intersecting a specific date. In a new script, paste the code below:

    +

    var palettes = require(‘users/gena/packages:palettes’);

    +

    var resultsPath = ‘projects/gee-book/assets/F4-7/Rondonia_example_small’;
    +var ccdResults = ee.Image(resultsPath);
    Map.centerObject(ccdResults, 10);
    print(ccdResults);

    -

    // Display segment start and end times.
    -var start = ccdResults.select(‘tStart’);
    -var end = ccdResults.select(‘tEnd’);
    -Map.addLayer(start, {
    -   min: 1999,
    -   max: 2001}, ‘Segment start’);
    -Map.addLayer(end, {
    -   min: 2010,
    -   max: 2020}, ‘Segment end’);

    -

    Check the Console and expand the bands section in the printed image information. We will be using the tStart, tEnd, and SWIR1_coefs bands, which are array images containing the date when the time segments start, date time segments end, and the coefficients for each of those segments for the SWIR1 band. Run the code above and switch the map to Satellite mode. Using the Inspector, click anywhere on the images, noticing the number of dates printed and their values for multiple clicked pixels. You will notice that for places with stable forest cover, there is usually one value for tStart and one for tEnd. This means that for those more stable places, only one time segment was fit by CCDC. On the other hand, for places with visible transformation in the basemap, the number of dates is usually two or three, meaning that the algorithm fitted two or three time segments, respectively. To simplify the processing of the data, we can select a single segment to extract its coefficients. Paste the code below and re-run the script:

    -

    // Find the segment that intersects a given date.
    -var targetDate = 2005.5;
    -var selectSegment = start.lte(targetDate).and(end.gt(targetDate));
    -Map.addLayer(selectSegment, {}, ‘Identified segment’);

    -

    In the code above, we set a time of interest, in this case the middle of 2005, and then we find the segments that meet the condition of starting before and ending after that date. Using the Inspector again, click on different locations and verify the outputs. The segment that meets the condition will have a value of 1, and the other segments will have a value of 0. We can use this information to select the coefficients for that segment, using the code below:

    -

    // Get all coefs in the SWIR1 band.
    -var SWIR1Coefs = ccdResults.select(‘SWIR1_coefs’);
    -Map.addLayer(SWIR1Coefs, {}, ‘SWIR1 coefs’);

    -

    // Select only those for the segment that we identified previously.
    -var sliceStart = selectSegment.arrayArgmax().arrayFlatten([
    -   [‘index’]
    -]);
    -var sliceEnd = sliceStart.add(1);
    -var selectedCoefs = SWIR1Coefs.arraySlice(0, sliceStart, sliceEnd);
    -Map.addLayer(selectedCoefs, {}, ‘Selected SWIR1 coefs’);

    -

    In the piece of code above, we first select the array image with the coefficients for the SWIR1 band. Then, using the layer that we created before, we find the position where the condition is true, and use that to extract the coefficients only for that segment. Once again, you can verify that using the Inspector tab.

    -

    Finally, what we have now is the full set of coefficients for the segment that intersects the midpoint of 2005. The coefficients are in the following order: intercept, slope, cosine 1, sine 1, cosine 2, sine 2, cosine 3, and sine 3. For this exercise we will extract the intercept coefficient (Fig. 4.7.4), which is the first element in the array, using the code below:

    -

    // Retrieve only the intercept coefficient.
    -var intercept = selectedCoefs.arraySlice(1, 0, 1).arrayProject([1]);
    -var intVisParams = {
    -   palette: palettes.matplotlib.viridis[7],
    -   min: -6,
    -   max: 6
    -};
    -Map.addLayer(intercept.arrayFlatten([
    -   [‘INTP’]
    -]), intVisParams, ‘INTP_SWIR1’);

    -

    -

    Fig. F4.7.4 Values for the intercept coefficient of the segments that start before and end after the midpoint of 2005

    +
    // Display segment start and end times.  
+var start = ccdResults.select('tStart');  
+var end = ccdResults.select('tEnd');  
+Map.addLayer(start, {  
+   min: 1999,  
+   max: 2001}, 'Segment start');  
+Map.addLayer(end, {  
+   min: 2010,  
+   max: 2020}, 'Segment end');
    +

    Check the Console and expand the bands section in the printed image information. We will be using the tStart, tEnd, and SWIR1_coefs bands, which are array images containing the date when the time segments start, date time segments end, and the coefficients for each of those segments for the SWIR1 band. Run the code above and switch the map to Satellite mode. Using the Inspector, click anywhere on the images, noticing the number of dates printed and their values for multiple clicked pixels. You will notice that for places with stable forest cover, there is usually one value for tStart and one for tEnd. This means that for those more stable places, only one time segment was fit by CCDC. On the other hand, for places with visible transformation in the basemap, the number of dates is usually two or three, meaning that the algorithm fitted two or three time segments, respectively. To simplify the processing of the data, we can select a single segment to extract its coefficients. Paste the code below and re-run the script:

    +
    // Find the segment that intersects a given date.  
+var targetDate = 2005.5;  
+var selectSegment = start.lte(targetDate).and(end.gt(targetDate));  
+Map.addLayer(selectSegment, {}, 'Identified segment');
    +

    In the code above, we set a time of interest, in this case the middle of 2005, and then we find the segments that meet the condition of starting before and ending after that date. Using the Inspector again, click on different locations and verify the outputs. The segment that meets the condition will have a value of 1, and the other segments will have a value of 0. We can use this information to select the coefficients for that segment, using the code below:

    +
    // Get all coefs in the SWIR1 band.  
+var SWIR1Coefs = ccdResults.select('SWIR1_coefs');  
+Map.addLayer(SWIR1Coefs, {}, 'SWIR1 coefs');  
+  
+// Select only those for the segment that we identified previously.  
+var sliceStart = selectSegment.arrayArgmax().arrayFlatten([  
+   ['index']  
+]);  
+var sliceEnd = sliceStart.add(1);  
+var selectedCoefs = SWIR1Coefs.arraySlice(0, sliceStart, sliceEnd);  
+Map.addLayer(selectedCoefs, {}, 'Selected SWIR1 coefs');
    +

    In the piece of code above, we first select the array image with the coefficients for the SWIR1 band. Then, using the layer that we created before, we find the position where the condition is true, and use that to extract the coefficients only for that segment. Once again, you can verify that using the Inspector tab.

    +

    Finally, what we have now is the full set of coefficients for the segment that intersects the midpoint of 2005. The coefficients are in the following order: intercept, slope, cosine 1, sine 1, cosine 2, sine 2, cosine 3, and sine 3. For this exercise we will extract the intercept coefficient (Fig. 4.7.4), which is the first element in the array, using the code below:

    +
    // Retrieve only the intercept coefficient.  
+var intercept = selectedCoefs.arraySlice(1, 0, 1).arrayProject([1]);  
+var intVisParams = {  
+   palette: palettes.matplotlib.viridis[7],  
+   min: -6,  
+   max: 6  
+};  
+Map.addLayer(intercept.arrayFlatten([  
+   ['INTP']  
+]), intVisParams, 'INTP_SWIR1');
    +
    +
    +

    +

    Fig. F4.7.4 Values for the intercept coefficient of the segments that start before and end after the midpoint of 2005

    +
    +

    Since we run the CCDC algorithm on Landsat surface reflectance images, intercept values should represent the average reflectance of a segment. However, if you click on the image, you will see that the values are outside of the 0–1 range. This is because the intercept is calculated by the CCDC algorithm for the origin (e.g., time 0), and not for the year we requested. In order to retrieve the adjusted intercept, as well as other coefficients, we will use a different approach.

    @@ -2268,83 +2683,101 @@ Note
    -

    Code Checkpoint F47d. The book’s repository contains a script that shows what your code should look like at this point.

    +

    Code Checkpoint F47d. The book’s repository contains a script that shows what your code should look like at this point.

    Section 5. Extracting Coefficients Using External Functions

    The code we generated in the previous section allowed us to extract a single coefficient for a single date. However, we typically want to extract a set of multiple coefficients and bands that we can use as inputs to other workflows, such as classification. To simplify that process, we will use the same function library that we saw in Sect. 2. In this section we will extract and visualize different coefficients for a single date and produce an RGB image using the intercept coefficients for multiple spectral bands for the same date. The first step involves determining the date of interest and converting the CCDC results from array images to regular multiband images for easier manipulation and faster display. In a new script, copy the code below:

    -

    // Load the required libraries.
    -var palettes = require(‘users/gena/packages:palettes’);
    -var utils = require(   ‘users/parevalo_bu/gee-ccdc-tools:ccdcUtilities/api’);

    -

    // Load the results.
    -var resultsPath =   ‘projects/gee-book/assets/F4-7/Rondonia_example_small’;
    -var ccdResults = ee.Image(resultsPath);
    -Map.centerObject(ccdResults, 10);

    -

    // Convert a date into fractional years.
    -var inputDate = ‘2005-09-25’;
    -var dateParams = {
    -   inputFormat: 3,
    -   inputDate: inputDate,
    -   outputFormat: 1
    -};
    -var formattedDate = utils.Dates.convertDate(dateParams);

    -

    // Band names originally used as inputs to the CCD algorithm.
    -var BANDS = [‘BLUE’, ‘GREEN’, ‘RED’, ‘NIR’, ‘SWIR1’, ‘SWIR2’];

    -

    // Names for the time segments to retrieve.
    -var SEGS = [‘S1’, ‘S2’, ‘S3’, ‘S4’, ‘S5’, ‘S6’, ‘S7’, ‘S8’, ‘S9’,   ‘S10’
    -];

    -

    // Transform CCD results into a multiband image.
    -var ccdImage = utils.CCDC.buildCcdImage(ccdResults, SEGS.length,
    -   BANDS);
    -print(ccdImage);

    +
    // Load the required libraries.  
+var palettes = require('users/gena/packages:palettes');  
+var utils = require(   'users/parevalo_bu/gee-ccdc-tools:ccdcUtilities/api');  
+  
+// Load the results.  
+var resultsPath =   'projects/gee-book/assets/F4-7/Rondonia_example_small';  
+var ccdResults = ee.Image(resultsPath);  
+Map.centerObject(ccdResults, 10);  
+  
+// Convert a date into fractional years.  
+var inputDate = '2005-09-25';  
+var dateParams = {  
+   inputFormat: 3,  
+   inputDate: inputDate,  
+   outputFormat: 1  
+};  
+var formattedDate = utils.Dates.convertDate(dateParams);  
+  
+// Band names originally used as inputs to the CCD algorithm.  
+var BANDS = ['BLUE', 'GREEN', 'RED', 'NIR', 'SWIR1', 'SWIR2'];  
+  
+// Names for the time segments to retrieve.  
+var SEGS = ['S1', 'S2', 'S3', 'S4', 'S5', 'S6', 'S7', 'S8', 'S9',   'S10'  
+];  
+  
+// Transform CCD results into a multiband image.  
+var ccdImage = utils.CCDC.buildCcdImage(ccdResults, SEGS.length,  
+   BANDS);  
+print(ccdImage);

    In the code above we define the date of interest (2005-09-25) and convert it to the date format in which we ran CCDC, which corresponds to fractional years. After that, we specify the band that we used as inputs for the CCDC algorithm. Finally, we specify the names we will assign to the time segments, with the list length indicating the maximum number of time segments to retrieve per pixel. This step is done because the results generated by CCDC are stored as variable-length arrays. For example, a pixel where there are no breaks detected will have one time segment, but another pixel where a single break was detected may have one or two segments, depending on when the break occurred. Requesting a pre-defined maximum number of segments ensures that the structure of the multi-band image is known, and greatly facilitates its manipulation and display. Once we have set these variables, we call a function that converts the result into an image with several bands representing the combination of segments requested, input bands, and coefficients. You can see the image structure in the Console.

    Finally, to extract a subset of coefficients for the desired bands, we can use a function in the imported library, called getMultiCoefs. This function expects the following ordered parameters:

    • The CCDC results in the multiband format we just generated in the step above.
    • The date for which we want to extract the coefficients, in the format in which the CCDC results were run (fractional years in our case).
    • List of the bands to retrieve (i.e., spectral bands).
      • -
    • List of coefficients to retrieve, defined as follows: INTP (intercept), SLP (slope), COS, SIN,COS32, SIN2, COS3, SIN3, and RMSE.
      • -
    • A Boolean flag of true or false, indicating whether we want the intercepts to be calculated for the input date, instead of being calculated at the origin. If true, SLP must be included in the list of coefficients to retrieve.
      • +
    • List of coefficients to retrieve, defined as follows: INTP (intercept), SLP (slope), COS, SIN,COS32, SIN2, COS3, SIN3, and RMSE.
      • +
    • A Boolean flag of true or false, indicating whether we want the intercepts to be calculated for the input date, instead of being calculated at the origin. If true, SLP must be included in the list of coefficients to retrieve.
    • List of segment names, as used to create the multiband image in the prior step.
      • -
    • Behavior to apply if there is no time segment for the requested date: normal will retrieve a value only if the date intersects a segment;        before or after will use the value of the segment immediately before or after the requested date, if no segment intersects the date directly.
      • +
    • Behavior to apply if there is no time segment for the requested date: normal will retrieve a value only if the date intersects a segment; before or after will use the value of the segment immediately before or after the requested date, if no segment intersects the date directly.
    -

    // Define bands to select.
    -var SELECT_BANDS = [‘RED’, ‘GREEN’, ‘BLUE’, ‘NIR’];

    -

    // Define coefficients to select.
    -// This list contains all possible coefficients, and the RMSE
    -var SELECT_COEFS = [‘INTP’, ‘SLP’, ‘RMSE’];

    -

    // Obtain coefficients.
    -var coefs = utils.CCDC.getMultiCoefs(
    -   ccdImage, formattedDate, SELECT_BANDS, SELECT_COEFS, true,
    -   SEGS, ‘after’);
    -print(coefs);

    -

    // Show a single coefficient.
    -var slpVisParams = {
    -   palette: palettes.matplotlib.viridis[7],
    -   min: -0.0005,
    -   max: 0.005
    -};
    -Map.addLayer(coefs.select(‘RED_SLP’), slpVisParams,   ‘RED SLOPE 2005-09-25’);

    -

    var rmseVisParams = {
    -   palette: palettes.matplotlib.viridis[7],
    -   min: 0,
    -   max: 0.1
    -};
    -Map.addLayer(coefs.select(‘NIR_RMSE’), rmseVisParams,   ‘NIR RMSE 2005-09-25’);

    -

    // Show an RGB with three coefficients.
    -var rgbVisParams = {
    -   bands: [‘RED_INTP’, ‘GREEN_INTP’, ‘BLUE_INTP’],
    -   min: 0,
    -   max: 0.1
    -};
    -Map.addLayer(coefs, rgbVisParams, ‘RGB 2005-09-25’);

    -

    The slope and RMSE images are shown in Fig. 4.7.5. For the slopes, high positive values are bright, while large negative values are very dark. Most of the remaining forest is stable and has a slope close to zero, while areas that have experienced transformation and show agricultural activity tend to have positive slopes in the RED band, appearing bright in the image. Similarly, for the RMSE image, stable forests present more predictable time series of surface reflectance that are captured more faithfully by the time segments, and therefore present lower RMSE values, appearing darker in the image. Agricultural areas present noisier time series that are more challenging to model, and result in higher RMSE values, appearing brighter.

    +
    // Define bands to select.  
+var SELECT_BANDS = ['RED', 'GREEN', 'BLUE', 'NIR'];  
+  
+// Define coefficients to select.  
+// This list contains all possible coefficients, and the RMSE  
+var SELECT_COEFS = ['INTP', 'SLP', 'RMSE'];  
+  
+// Obtain coefficients.  
+var coefs = utils.CCDC.getMultiCoefs(  
+   ccdImage, formattedDate, SELECT_BANDS, SELECT_COEFS, true,  
+   SEGS, 'after');  
+print(coefs);  
+  
+// Show a single coefficient.  
+var slpVisParams = {  
+   palette: palettes.matplotlib.viridis[7],  
+   min: -0.0005,  
+   max: 0.005  
+};  
+Map.addLayer(coefs.select('RED_SLP'), slpVisParams,   'RED SLOPE 2005-09-25');  
+  
+var rmseVisParams = {  
+   palette: palettes.matplotlib.viridis[7],  
+   min: 0,  
+   max: 0.1  
+};  
+Map.addLayer(coefs.select('NIR_RMSE'), rmseVisParams,   'NIR RMSE 2005-09-25');  
+  
+// Show an RGB with three coefficients.  
+var rgbVisParams = {  
+   bands: ['RED_INTP', 'GREEN_INTP', 'BLUE_INTP'],  
+   min: 0,  
+   max: 0.1  
+};  
+Map.addLayer(coefs, rgbVisParams, 'RGB 2005-09-25');
    +

    The slope and RMSE images are shown in Fig. 4.7.5. For the slopes, high positive values are bright, while large negative values are very dark. Most of the remaining forest is stable and has a slope close to zero, while areas that have experienced transformation and show agricultural activity tend to have positive slopes in the RED band, appearing bright in the image. Similarly, for the RMSE image, stable forests present more predictable time series of surface reflectance that are captured more faithfully by the time segments, and therefore present lower RMSE values, appearing darker in the image. Agricultural areas present noisier time series that are more challenging to model, and result in higher RMSE values, appearing brighter.

    -

    -

    Fig. F4.7.5 Image showing the slopes (top) and RMSE (bottom) of the segments that intersect the requested date

    -

    Finally, the RGB image we created is shown in Fig. 4.7.6. The intercepts are calculated for the middle point of the time segment intercepting the date we requested, representing the average reflectance for the span of the selected segment. In that sense, when shown together as an RGB image, they are similar to a composite image for the selected date, with the advantage of always being cloud-free.

    -

    -

    Fig. F4.7.6 RGB image created using the time segment intercepts for the requested date

    +
    +
    +

    +

    Fig. F4.7.5 Image showing the slopes (top) and RMSE (bottom) of the segments that intersect the requested date

    +
    +
    +

    Finally, the RGB image we created is shown in Fig. 4.7.6. The intercepts are calculated for the middle point of the time segment intercepting the date we requested, representing the average reflectance for the span of the selected segment. In that sense, when shown together as an RGB image, they are similar to a composite image for the selected date, with the advantage of always being cloud-free.

    +
    +
    +

    +

    Fig. F4.7.6 RGB image created using the time segment intercepts for the requested date

    +
    +
    @@ -2355,7 +2788,7 @@ Note
    -

    Code Checkpoint F47e. The book’s repository contains a script that shows what your code should look like at this point.

    +

    Code Checkpoint F47e. The book’s repository contains a script that shows what your code should look like at this point.

    @@ -2363,11 +2796,11 @@ Note

    Synthesis

    Assignment 1. Use the time series from the first section of this chapter to explore the time series and time segments produced by CCDC in many locations around the world. Compare places with different land cover types, and places with more stable dynamics (e.g., lakes, primary forests) vs. highly dynamic places (e.g., agricultural lands, construction sites). Pay attention to the variability in data density across continents and latitudes, and the effect that data density has on the appearance of the time segments. Use different spectral bands and indices and notice how they capture the temporal dynamics you are observing.

    Assignment 2. Pick three periods within the temporal study period of the CCDC results we used earlier: one near to the start, another in the middle, and the third close to the end. For each period, visualize the maximum change magnitude. Compare the spatial patterns between periods, and reflect on the types of disturbances that might be happening at each stage.

    -

    Assignment 3. Select the intercept coefficients of the middle date of each of the periods you chose in the previous assignment. For each of those dates, load an RGB image with the band combination of your choosing (or simply use the Red, Green and Blue intercepts to obtain true-color images). Using the Inspector tab, compare the values across images in places with subtle and large differences between them, as well as in areas that do not change. What do the values tell you in terms of the benefits of using CCDC to study changes in a landscape?

    +

    Assignment 3. Select the intercept coefficients of the middle date of each of the periods you chose in the previous assignment. For each of those dates, load an RGB image with the band combination of your choosing (or simply use the Red, Green and Blue intercepts to obtain true-color images). Using the Inspector tab, compare the values across images in places with subtle and large differences between them, as well as in areas that do not change. What do the values tell you in terms of the benefits of using CCDC to study changes in a landscape?

    Conclusion

    -

    This chapter provided a guide for the interpretation of the results from the CCDC algorithm for studying deforestation in the Amazon. Consider the advantages of such an analysis compared to traditional approaches to change detection, which are typically based on the comparison of two or a few images collected over the same area. For example, with time-series analysis, we can study trends and subtle processes such as vegetation recovery or degradation, determine the timing of land-surface events, and move away from retrospective analyses to monitoring in near-real time. Through the use of all available clear observations, CCDC can detect intra-annual breaks and capture seasonal patterns, although at the expense of increased computational requirements and complexity, unlike faster and easier to interpret methods based on annual composites, such as LandTrendr (Chap. F4.5). We expect to see more applications that make use of multiple change detection approaches (also known as “Ensemble” approaches), and multisensor analyses in which data from different satellites are fused (radar and optical, for example) for higher data density.

    +

    This chapter provided a guide for the interpretation of the results from the CCDC algorithm for studying deforestation in the Amazon. Consider the advantages of such an analysis compared to traditional approaches to change detection, which are typically based on the comparison of two or a few images collected over the same area. For example, with time-series analysis, we can study trends and subtle processes such as vegetation recovery or degradation, determine the timing of land-surface events, and move away from retrospective analyses to monitoring in near-real time. Through the use of all available clear observations, CCDC can detect intra-annual breaks and capture seasonal patterns, although at the expense of increased computational requirements and complexity, unlike faster and easier to interpret methods based on annual composites, such as LandTrendr (Chap. F4.5). We expect to see more applications that make use of multiple change detection approaches (also known as “Ensemble” approaches), and multisensor analyses in which data from different satellites are fused (radar and optical, for example) for higher data density.

    References

    @@ -2378,7 +2811,6 @@ Note

    Woodcock CE, Loveland TR, Herold M, Bauer ME (2020) Transitioning from change detection to monitoring with remote sensing: A paradigm shift. Remote Sens Environ 238:111558. https://doi.org/10.1016/j.rse.2019.111558

    Zhu Z, Woodcock CE (2014) Continuous change detection and classification of land cover using all available Landsat data. Remote Sens Environ 144:152–171. https://doi.org/10.1016/j.rse.2014.01.011

    -

    14 Data Fusion: Merging Classification Streams

    @@ -2397,8 +2829,7 @@ Chapter Information

    Overview

    -

     

    -

    As the ability to rapidly produce classifications of satellite images grows, it will be increasingly important to have algorithms that can sift through them to separate the signal from inevitable classification noise. The purpose of this chapter is to explore how to update classification time series by blending information from multiple classifications made from a wide variety of data sources. In this lab, we will explore how to update the classification time series of the Roosevelt River found in Fortin et al. (2020). That time series began with the 1972 launch of Landsat 1, blending evidence from 10 sensors and more than 140 images to show the evolution of the area until 2016. How has it changed since 2016? What new tools and data streams might we tap to understand the land surface through time?

    +

    As the ability to rapidly produce classifications of satellite images grows, it will be increasingly important to have algorithms that can sift through them to separate the signal from inevitable classification noise. The purpose of this chapter is to explore how to update classification time series by blending information from multiple classifications made from a wide variety of data sources. In this lab, we will explore how to update the classification time series of the Roosevelt River found in Fortin et al. (2020). That time series began with the 1972 launch of Landsat 1, blending evidence from 10 sensors and more than 140 images to show the evolution of the area until 2016. How has it changed since 2016? What new tools and data streams might we tap to understand the land surface through time?

    Learning Outcomes

    @@ -2413,23 +2844,25 @@ Chapter Information
    • Import images and image collections, filter, and visualize (Part F1).
    • Perform basic image analysis: select bands, compute indices, create masks, classify images (Part F2).
      • -
    • ​​Create a graph using ui.Chart (Chap. F1.3).
      • +
    • ​​Create a graph using ui.Chart (Chap. F1.3).
    • Obtain accuracy metrics from classifications (Chap. F2.2).
    + +

    Introduction

    When working with multiple sensors, we are often presented with a challenge: What to do with classification noise? It’s almost impossible to remove all noise from a classification. Given the information contained in a stream of classifications, however, you should be able to use the temporal context to distinguish noise from true changes in the landscape.

    -

    The Bayesian Updating of Land Cover (BULC) algorithm (Cardille and Fortin 2016) is designed to extract the signal from the noise in a stream of classifications made from any number of data sources. BULC’s principal job is to estimate, at each time step, the likeliest state of land use and land cover (LULC) in a study area given the accumulated evidence to that point. It takes a stack of provisional classifications as input; in keeping with the terminology of Bayesian statistics, these are referred to as “Events,” because they provide new evidence to the system. BULC then returns a stack of classifications as output that represents the estimated LULC time series implied by the Events. 

    +

    The Bayesian Updating of Land Cover (BULC) algorithm (Cardille and Fortin 2016) is designed to extract the signal from the noise in a stream of classifications made from any number of data sources. BULC’s principal job is to estimate, at each time step, the likeliest state of land use and land cover (LULC) in a study area given the accumulated evidence to that point. It takes a stack of provisional classifications as input; in keeping with the terminology of Bayesian statistics, these are referred to as “Events,” because they provide new evidence to the system. BULC then returns a stack of classifications as output that represents the estimated LULC time series implied by the Events.

    BULC estimates, at each time step, the most likely class from a set given the evidence up to that point in time. This is done by employing an accuracy assessment matrix like that seen in Chap. F2.2. At each time step, the algorithm quantifies the agreement between two classifications adjacent in time within a time series.

    -

    If the Events agree strongly, they are evidence of the true condition of the landscape at that point in time. If two adjacent Events disagree, the accuracy assessment matrix limits their power to change the class of a pixel in the interpreted time series. As each new classification is processed, BULC judges the credibility of a pixel’s stated class and keeps track of a set of estimates of the probability of each class for each pixel. In this way, each pixel traces its own LULC history, reflected through BULC’s judgment of the confidence in each of the classifications. The specific mechanics and formulas of BULC are detailed in Cardille and Fortin (2016).

    -

    BULC’s code is written in JavaScript, with modules that weigh evidence for and against change in several ways, while recording parts of the data-weighing process for you to inspect. In this lab, we will explore BULC through its graphical user interface (GUI), which allows rapid interaction with the algorithm’s main functionality.

    +

    If the Events agree strongly, they are evidence of the true condition of the landscape at that point in time. If two adjacent Events disagree, the accuracy assessment matrix limits their power to change the class of a pixel in the interpreted time series. As each new classification is processed, BULC judges the credibility of a pixel’s stated class and keeps track of a set of estimates of the probability of each class for each pixel. In this way, each pixel traces its own LULC history, reflected through BULC’s judgment of the confidence in each of the classifications. The specific mechanics and formulas of BULC are detailed in Cardille and Fortin (2016).

    +

    BULC’s code is written in JavaScript, with modules that weigh evidence for and against change in several ways, while recording parts of the data-weighing process for you to inspect. In this lab, we will explore BULC through its graphical user interface (GUI), which allows rapid interaction with the algorithm’s main functionality.

    14.1 Imagery and Classifications of the Roosevelt River

    -

    How has the Roosevelt River area changed in recent decades? One way to view the area’s recent history is to use Google Earth Timelapse, which shows selected annual clear images of every part of Earth’s terrestrial surface since the 1980s. (You can find the site quickly with a web search.) Enter “Roosevelt River, Brazil” in the search field. For centuries, this area was very remote from agricultural development. It was so little known to Westerners that when former US President Theodore Roosevelt traversed it in the early 1900s there was widespread doubt about whether his near-death experience there was exaggerated or even entirely fictional (Millard 2006). After World War II, the region saw increased agricultural development. Fortin et al. (2020) traced four decades of the history of this region with satellite imagery. Timelapse, meanwhile, indicates that land cover conversion continued after 2016. Can we track it using Earth Engine?

    -

    In this section, we will view the classification inputs to BULC, which were made separately from this lab exercise by identifying training points and classifying them using Earth Engine’s regression tree capability. As seen in Table 4.8.1, the classification inputs included Sentinel-2 optical data, Landsat 7, Landsat 8, and the Advanced Spaceborne Thermal Emission and Reflection Radiometer (ASTER) aboard Terra. Though each classification was made with care, they each contain noise, with each pixel likely to have been misclassified one or more times. This could lead us to draw unrealistic conclusions if the classifications themselves were considered as a time series. For example, we would judge it highly unlikely that an area represented by a pixel would really be agriculture one day and revert to intact forest later in the month, only to be converted to agriculture again soon after, and so on. With careful (though unavoidably imperfect) classifications, we would expect that an area that had truly been converted to agriculture would consistently be classified as agriculture, while an area that remained as forest would be classified as that class most of the time. BULC’s logic is to detect that persistence, extracting the true LULC change and stability from the noisy signal of the time series of classifications.

    -

    Table F4.8.1 Images classified for updating Roosevelt River LULC with BULC

    +

    How has the Roosevelt River area changed in recent decades? One way to view the area’s recent history is to use Google Earth Timelapse, which shows selected annual clear images of every part of Earth’s terrestrial surface since the 1980s. (You can find the site quickly with a web search.) Enter “Roosevelt River, Brazil” in the search field. For centuries, this area was very remote from agricultural development. It was so little known to Westerners that when former US President Theodore Roosevelt traversed it in the early 1900s there was widespread doubt about whether his near-death experience there was exaggerated or even entirely fictional (Millard 2006). After World War II, the region saw increased agricultural development. Fortin et al. (2020) traced four decades of the history of this region with satellite imagery. Timelapse, meanwhile, indicates that land cover conversion continued after 2016. Can we track it using Earth Engine?

    +

    In this section, we will view the classification inputs to BULC, which were made separately from this lab exercise by identifying training points and classifying them using Earth Engine’s regression tree capability. As seen in Table 4.8.1, the classification inputs included Sentinel-2 optical data, Landsat 7, Landsat 8, and the Advanced Spaceborne Thermal Emission and Reflection Radiometer (ASTER) aboard Terra. Though each classification was made with care, they each contain noise, with each pixel likely to have been misclassified one or more times. This could lead us to draw unrealistic conclusions if the classifications themselves were considered as a time series. For example, we would judge it highly unlikely that an area represented by a pixel would really be agriculture one day and revert to intact forest later in the month, only to be converted to agriculture again soon after, and so on. With careful (though unavoidably imperfect) classifications, we would expect that an area that had truly been converted to agriculture would consistently be classified as agriculture, while an area that remained as forest would be classified as that class most of the time. BULC’s logic is to detect that persistence, extracting the true LULC change and stability from the noisy signal of the time series of classifications.

    +

    Table F4.8.1 Images classified for updating Roosevelt River LULC with BULC

    Sensor

    Date

    Spatial resolution

    @@ -2453,16 +2886,16 @@ Chapter Information

    2019: June 19

    2020: August 8

    15m–30m

    -

    As you have seen in earlier chapters, creating classifications can be very involved and time consuming. To allow you to concentrate on BULC’s efforts to clean noise from an existing ImageCollection, we have created the classifications already and stored them as an ImageCollection asset. You can view the Event time series using the ui.Thumbnail function, which creates an animation of the elements of the collection​​​​. Paste the code below into a new script to see those classifications drawn in sequence in the Console.

    -

    var events = ee.ImageCollection(   ‘projects/gee-book/assets/F4-8/cleanEvents’);
    +

    As you have seen in earlier chapters, creating classifications can be very involved and time consuming. To allow you to concentrate on BULC’s efforts to clean noise from an existing ImageCollection, we have created the classifications already and stored them as an ImageCollection asset. You can view the Event time series using the ui.Thumbnail function, which creates an animation of the elements of the collection​​​​. Paste the code below into a new script to see those classifications drawn in sequence in the Console.

    +

    var events = ee.ImageCollection( ‘projects/gee-book/assets/F4-8/cleanEvents’);
    print(events, ‘List of Events’);
    print(‘Number of events:’, events.size());

    print(ui.Thumbnail(events, {
    -   min: 0,
    -   max: 3,
    -   palette: [‘black’, ‘green’, ‘blue’, ‘yellow’],
    -   framesPerSecond: 1,
    -   dimensions: 1000
    +min: 0,
    +max: 3,
    +palette: [‘black’, ‘green’, ‘blue’, ‘yellow’],
    +framesPerSecond: 1,
    +dimensions: 1000
    }));

    In the thumbnail sequence, the color palette shows Forest (class 1) as green, Water (class 2) as blue, and Active Agriculture (class 3) as yellow. Areas with no data in a particular Event are shown in black.

    @@ -2475,69 +2908,85 @@ Note
    -

    Code Checkpoint F48a. The book’s repository contains a script that shows what your code should look like at this point.

    +

    Code Checkpoint F48a. The book’s repository contains a script that shows what your code should look like at this point.

    14.2 Basics of the BULC Interface

    -

    To see if BULC can successfully sift through these Events, we will use BULC’s GUI (Fig. F4.8.1), which makes interacting with the functionality straightforward. ::: {.callout-note} Code Checkpoint F48b in the book’s repository contains information about accessing that interface.

    +

    To see if BULC can successfully sift through these Events, we will use BULC’s GUI (Fig. F4.8.1), which makes interacting with the functionality straightforward. :::{.callout-note} Code Checkpoint F48b in the book’s repository contains information about accessing that interface.

    +
    -

    -

    Fig. F4.8.1 BULC interface

    -

    After you have run the script, BULC’s interface requires that a few parameters be set; these are specified using the left panel. Here, we describe and populate each of the required parameters, which are shown in red. As you proceed, the default red color will change to green when a parameter receives a value.

    +
    +
    +

    +

    Fig. F4.8.1 BULC interface

    +
    +
    +

    After you have run the script, BULC’s interface requires that a few parameters be set; these are specified using the left panel. Here, we describe and populate each of the required parameters, which are shown in red. As you proceed, the default red color will change to green when a parameter receives a value.

      -
    • The interface permits new runs to be created using the Manual or Automated methods. The Automated setting allows information from a previous run to be used without manual entry. In this tutorial, we will enter each parameter individually using the interface, so you should set this item to Manual by clicking once on it.
      • -
    • Select type of image: The interface can accept pre-made Event inputs in one of three forms: (1) as a stored ImageCollection; (2) as a single multi-banded Image; and (3) as a stream of Dynamic World classifications. The classifications are processed in the order they are given, either within the ImageCollection or sequentially through the Image bands. For this run, select Image Collection from the dropdown menu, then enter the path to this collection, without enclosing it in quotes: projects/gee-book/assets/F4-8/cleanEvents
      • +
    • The interface permits new runs to be created using the Manual or Automated methods. The Automated setting allows information from a previous run to be used without manual entry. In this tutorial, we will enter each parameter individually using the interface, so you should set this item to Manual by clicking once on it.
      • +
    • Select type of image: The interface can accept pre-made Event inputs in one of three forms: (1) as a stored ImageCollection; (2) as a single multi-banded Image; and (3) as a stream of Dynamic World classifications. The classifications are processed in the order they are given, either within the ImageCollection or sequentially through the Image bands. For this run, select Image Collection from the dropdown menu, then enter the path to this collection, without enclosing it in quotes: projects/gee-book/assets/F4-8/cleanEvents
    • Remap: in some settings, you might want to remap the input value to combine classes. Leave this empty for now; an example of this is discussed later in the lab.
      • -
    • Number of Classes in Events and Number of Classes to Track: The algorithm requires the number of classes in each Event and the number of meaningful classes to track to be entered. Here, there are 3 classes in each classification (Forest, Water, and Active Agriculture) and 3 classes being tracked. (In the BULC-U version of the algorithm (Lee et al. 2018, 2020), these numbers may be different when the Events are made using unsupervised classifications, which may contain many more classes than are being tracked in a given run.) Meaningful classes are assumed by BULC to begin with 1 rather than 0, while class values of 0 in Events are treated as no data. As seen in the thumbnail of Events, there are 3 classes; set both of these values to 3.
      • -
    • The Default Study Area is used by BULC to delimit the location to analyze. This value can be pulled from a specially sized Image or set automatically, using the extent of the inputs. Set this parameter to Event Geometry, which gets the value automatically from the Event collection.
      • -
    • The Base Land Cover Image defines the initial land cover condition to which BULC adds evidence from Events. Here, we are working to update the land cover map from the end of 2016, as estimated in Fortin et al. (2020). The ending estimated classification from that study has been loaded as an asset and placed as the first image in the input ImageCollection. We will direct the BULC interface to use this first image in the collection as the base land cover image by selecting Top.
      • -
    • Overlay Approach: BULC can run in multiple modes, which affect the outcome of the classification updating. One option, Overlay, overlays each consecutive Event with the one prior in the sequence, following Fortin et al. (2016). Another option, Custom, allows a user-defined constant array to be used. For this tutorial, we will choose D (Identity matrix), which uses the same transition table for every Event, regardless of how it overlays with the Event prior. That table gives a large conditional likelihood to the chance that classes agree strongly across the consecutive Event classifications that are used as inputs.
      • +
    • Number of Classes in Events and Number of Classes to Track: The algorithm requires the number of classes in each Event and the number of meaningful classes to track to be entered. Here, there are 3 classes in each classification (Forest, Water, and Active Agriculture) and 3 classes being tracked. (In the BULC-U version of the algorithm (Lee et al. 2018, 2020), these numbers may be different when the Events are made using unsupervised classifications, which may contain many more classes than are being tracked in a given run.) Meaningful classes are assumed by BULC to begin with 1 rather than 0, while class values of 0 in Events are treated as no data. As seen in the thumbnail of Events, there are 3 classes; set both of these values to 3.
      • +
    • The Default Study Area is used by BULC to delimit the location to analyze. This value can be pulled from a specially sized Image or set automatically, using the extent of the inputs. Set this parameter to Event Geometry, which gets the value automatically from the Event collection.
      • +
    • The Base Land Cover Image defines the initial land cover condition to which BULC adds evidence from Events. Here, we are working to update the land cover map from the end of 2016, as estimated in Fortin et al. (2020). The ending estimated classification from that study has been loaded as an asset and placed as the first image in the input ImageCollection. We will direct the BULC interface to use this first image in the collection as the base land cover image by selecting Top.
      • +
    • Overlay Approach: BULC can run in multiple modes, which affect the outcome of the classification updating. One option, Overlay, overlays each consecutive Event with the one prior in the sequence, following Fortin et al. (2016). Another option, Custom, allows a user-defined constant array to be used. For this tutorial, we will choose D (Identity matrix), which uses the same transition table for every Event, regardless of how it overlays with the Event prior. That table gives a large conditional likelihood to the chance that classes agree strongly across the consecutive Event classifications that are used as inputs.
    -

    BULC makes relatively small demands on memory since its arithmetic uses only multiplication, addition, and division, without the need for complex function fitting. The specific memory use is tied to the overlay method used. In particular, Event-by-Event comparisons (the Overlay setting) are considerably more computationally expensive than pre-defined transition tables (the Identity and Custom settings). The maximum working Event depth is also slightly lowered when intermediate probability values are returned for inspection. Our tests indicate that with pre-defined truth tables and no intermediate probability values returned, BULC can handle updating problems hundreds of Events deep across an arbitrarily large area.

    +

    BULC makes relatively small demands on memory since its arithmetic uses only multiplication, addition, and division, without the need for complex function fitting. The specific memory use is tied to the overlay method used. In particular, Event-by-Event comparisons (the Overlay setting) are considerably more computationally expensive than pre-defined transition tables (the Identity and Custom settings). The maximum working Event depth is also slightly lowered when intermediate probability values are returned for inspection. Our tests indicate that with pre-defined truth tables and no intermediate probability values returned, BULC can handle updating problems hundreds of Events deep across an arbitrarily large area.

      -
    • Initialization Approach: If a BULC run of the full intended size ever surpassed the memory available, you would be able to break the processing into two or more parts using the following technique. First, create a slightly smaller run that can complete, and save the final probability image of that run. Because of the operation of Bayes’ theorem, the ending probability multi-band image can be used as the prior probability to continue processing Events with the same answers as if it had been run all at once. For this small run, we will select F (First Run).
      • -
    • Levelers: BULC uses three levelers as part of its processing, as described in Fortin et al. (2016). The Initialization Leveler creates the initial probability vector of the initial LULC image; the Transition Leveler dampens transitions at each time step, making BULC less reactive to new evidence; and the Posterior Leveler ensures that each class retains nonzero probability so that the Bayes formula can function properly throughout the run. For this run, set the parameters to 0.65, 0.3, and 0.6, respectively. This corresponds to a typical set of values that is appropriate when moderate-quality classifications are fed to BULC.
      • -
    • Colour Output Palette: We will use the same color palette as what was seen in the small script you used to draw the Events, with one exception. Because BULC will give a value for the estimated class for every pixel, there are no pixels in the study area with missing or masked data. To line up the colors with the attainable numbers, we will remove the color ‘black’ from the specification. For this field, enter this list: [‘green’, ‘blue’, ‘yellow’]. For all of the text inputs, make sure to click outside that field after entering text so that the input information is registered; the changing of the text color to green confirms that the information was received.
      • +
    • Initialization Approach: If a BULC run of the full intended size ever surpassed the memory available, you would be able to break the processing into two or more parts using the following technique. First, create a slightly smaller run that can complete, and save the final probability image of that run. Because of the operation of Bayes’ theorem, the ending probability multi-band image can be used as the prior probability to continue processing Events with the same answers as if it had been run all at once. For this small run, we will select F (First Run).
      • +
    • Levelers: BULC uses three levelers as part of its processing, as described in Fortin et al. (2016). The Initialization Leveler creates the initial probability vector of the initial LULC image; the Transition Leveler dampens transitions at each time step, making BULC less reactive to new evidence; and the Posterior Leveler ensures that each class retains nonzero probability so that the Bayes formula can function properly throughout the run. For this run, set the parameters to 0.65, 0.3, and 0.6, respectively. This corresponds to a typical set of values that is appropriate when moderate-quality classifications are fed to BULC.
      • +
    • Colour Output Palette: We will use the same color palette as what was seen in the small script you used to draw the Events, with one exception. Because BULC will give a value for the estimated class for every pixel, there are no pixels in the study area with missing or masked data. To line up the colors with the attainable numbers, we will remove the color ‘black’ from the specification. For this field, enter this list: [‘green’, ‘blue’, ‘yellow’]. For all of the text inputs, make sure to click outside that field after entering text so that the input information is registered; the changing of the text color to green confirms that the information was received.

    When you have finished setting the required parameters, the interface will look like Fig. 4.8.2.

    -

    -

    Fig. 4.8.2 Initial settings for the key driving parameters of BULC 

    +
    +
    +

    +

    Fig. 4.8.2 Initial settings for the key driving parameters of BULC

    +
    +

    Beneath the required parameters is a set of optional parameters that affect which intermediate results are stored during a run for later inspection. We are also given a choice of returning intermediate results for closer inspection. At this stage, you can leave all optional parameters out of the BULC call by leaving them blanked or unchecked.

    -

    After clicking the Apply Parameters button at the bottom of the left panel, the classifications and parameters are sent to the BULC modules. The Map will move to the study area, and after a few seconds, the Console will hold new thumbnails. The uppermost thumbnail is a rapidly changing view of the input classifications. Beneath that is a thumbnail of the same area as interpreted by BULC. Beneath those is a Confidence thumbnail, which is discussed in detail later in this lab.

    -

    The BULC interpretation of the landscape looks roughly like the Event inputs, but it is different in two important ways. First, depending on the leveler settings, it will usually have less noise than the Event classifications. In the settings above, we used the Transition and Posterior levelers to tell BULC to trust past accumulated evidence more than a single new image. The second key difference between the BULC result and the input classifications is that even when the inputs don’t cover the whole area at each time step, BULC provides an estimate in every pixel at each time step. To create this continuous classification, if a new classification does not have data for some part of the study area (beyond the edge of a given image, for example), the last best guess from the previous iteration is carried forward. Simply put, the estimate in a given pixel is kept the same until new data arrives.

    -

    Meanwhile, below the Console, the rest of the interface changes when BULC is run. The Map panel displays BULC’s classification for the final date: that is, after considering the evidence from each of the input classifications. We can use the Satellite background to judge whether BULC is accurately capturing the state of LULC. This can be done by unselecting the drawn layers in the map layer set and selecting Satellite from the choices in the upper-right part of the Map panel. Earth Engine’s background satellite images are often updated, so you should see something like the right side of Fig. F4.8.3, though it may differ slightly. 

    +

    After clicking the Apply Parameters button at the bottom of the left panel, the classifications and parameters are sent to the BULC modules. The Map will move to the study area, and after a few seconds, the Console will hold new thumbnails. The uppermost thumbnail is a rapidly changing view of the input classifications. Beneath that is a thumbnail of the same area as interpreted by BULC. Beneath those is a Confidence thumbnail, which is discussed in detail later in this lab.

    +

    The BULC interpretation of the landscape looks roughly like the Event inputs, but it is different in two important ways. First, depending on the leveler settings, it will usually have less noise than the Event classifications. In the settings above, we used the Transition and Posterior levelers to tell BULC to trust past accumulated evidence more than a single new image. The second key difference between the BULC result and the input classifications is that even when the inputs don’t cover the whole area at each time step, BULC provides an estimate in every pixel at each time step. To create this continuous classification, if a new classification does not have data for some part of the study area (beyond the edge of a given image, for example), the last best guess from the previous iteration is carried forward. Simply put, the estimate in a given pixel is kept the same until new data arrives.

    +

    Meanwhile, below the Console, the rest of the interface changes when BULC is run. The Map panel displays BULC’s classification for the final date: that is, after considering the evidence from each of the input classifications. We can use the Satellite background to judge whether BULC is accurately capturing the state of LULC. This can be done by unselecting the drawn layers in the map layer set and selecting Satellite from the choices in the upper-right part of the Map panel. Earth Engine’s background satellite images are often updated, so you should see something like the right side of Fig. F4.8.3, though it may differ slightly.

    -

    -

    Fig. 4.8.3 BULC estimation of the state of LULC at the end of 2021 (left). Satellite backdrop for Earth Engine (right), which may differ from what you see due to updates.  

    -

    Question 1. When comparing the BULC classification for 2021 against the current Earth Engine satellite view, what are the similarities and differences? Note that in Earth Engine, the copyrighted year numbers at the bottom of the screen may not coincide with the precise date of the image shown.

    +
    +
    +

    +

    Fig. 4.8.3 BULC estimation of the state of LULC at the end of 2021 (left). Satellite backdrop for Earth Engine (right), which may differ from what you see due to updates.

    +
    +
    +

    Question 1. When comparing the BULC classification for 2021 against the current Earth Engine satellite view, what are the similarities and differences? Note that in Earth Engine, the copyrighted year numbers at the bottom of the screen may not coincide with the precise date of the image shown.

    In the rightmost panel below the Console, the interface offers you multiple options for viewing the results. These include:

      -
    1. Movie. This uses the ui.Thumbnail API function to draw the BULC results rapidly in the viewer.  This option offers you a control on the frame rate (in frames per second), and a checkbox affecting the drawing resolution. The high-resolution option uses the maximum resolution permitted given the function’s constraints. A lower resolution setting constructs the thumbnail more quickly, but at a loss of detail.
      2. +
    2. Movie. This uses the ui.Thumbnail API function to draw the BULC results rapidly in the viewer. This option offers you a control on the frame rate (in frames per second), and a checkbox affecting the drawing resolution. The high-resolution option uses the maximum resolution permitted given the function’s constraints. A lower resolution setting constructs the thumbnail more quickly, but at a loss of detail.
    3. Filmstrip. This produces an image like the Movie option, but allows you to move on request through each image.
      4. -
    4. Mosaic. This draws every BULC result in the panel. Depending on the size of the stack of classifications, this could become quite a large set of images.
      5. -
    5. Zoom. This draws the final BULC classification at multiple scales, with the finest-scale image matching that shown in the Map window.
      6. +
    6. Mosaic. This draws every BULC result in the panel. Depending on the size of the stack of classifications, this could become quite a large set of images.
      7. +
    7. Zoom. This draws the final BULC classification at multiple scales, with the finest-scale image matching that shown in the Map window.
    -

    Question 2. Select the BULC option, then select the Movie tool to view the result, and choose a drawing speed and resolution. When viewing the full area, would you assess the additional LULC changes since 2016 as being minor, moderate, or major compared to the changes that occurred before 2016? Explain the reasoning for your assessment.

    +

    Question 2. Select the BULC option, then select the Movie tool to view the result, and choose a drawing speed and resolution. When viewing the full area, would you assess the additional LULC changes since 2016 as being minor, moderate, or major compared to the changes that occurred before 2016? Explain the reasoning for your assessment.

    14.3 Detailed LULC Inspection with BULC

    -

    BULC results can be viewed interactively, allowing you to view more detailed estimations of the LULC around the study area. We will zoom into a specific area where change did occur after 2016. To do that, turn on the Satellite view and zoom in. Watching the scale bar in the lower right of the Map panel, continue zooming until the scale bar says 5 km. Then, enter “-60.742, -9.844” in the Earth Engine search tool, located above the code. The text will be interpreted as a longitude/latitude value and will offer you a nearby coordinate, indicated with a value for the degrees West and the degrees South. Click that entry and Earth Engine will move to that location, while keeping at the specified zoom level. Let’s compare the BULC result in this sector against the image from Earth Engine’s satellite view that is underneath it (Fig. 4.8.4).

    -

     

    -

    -

    Fig. 4.8.4 Comparison of the final classification of the northern part of the study area to the satellite view

    -

    BULC captured the changes between 2016 and 2021 with a classification series that suggests agricultural development (Fig. 4.8.4, left). Given the appearance of BULC’s 2021 classification, it suggests that the satellite backdrop at the time of this writing (Fig. 4.8.4, right) came from an earlier time period.

    -

    Now, in the Results panel, select BULC, then Movie. Set your desired frame speed and resolution, then select Redraw Thumbnail. Then, zoom the main Map even closer to some agriculture that appears to have been established between 2016 and 2021. Redraw the thumbnail movie as needed to find an interesting set of pixels.

    -

    With this finer-scale access to the results of BULC, you can select individual pixels to inspect. Move the horizontal divider downward to expose the Inspector tab and Console tab. Use the Inspector to click on several pixels to learn their history as expressed in the inputted Events and in BULC’s interpretation of the noise and signal in the Event series. In a chosen pixel, you might see output that looks like Fig. 4.8.5. It indicates a possible conversion in the Event time series after a few classifications of the pixel as Forest. This decreases the confidence that the pixel is still Forest (Fig. 4.8.5, lower panel), but not enough for the Active Agriculture class (class 3) to become the dominant probability. After the subsequent Event labels the pixel as Forest, the confidence (lower panel) recovers slightly, but not to its former level. The next Event classifies the pixel as Active Agriculture, confidently, by interpreting that second Active Agriculture classification, in a setting where change was already somewhat suspected after the first non-Forest classification. BULC’s label (middle panel) changes to be Active Agriculture at that point in the sequence. Subsequent Event classifications as Active Agriculture creates a growing confidence that its proper label at the end of the sequence was indeed Active Agriculture.

    -

    -

    Fig. 4.8.5 History for 2016-2020 for a pixel that appeared to have been newly cultivated during that period. (above): the input classifications, which s

    -

    Question 3. Run the code again with the same data, but adjust the three levelers, then view the results presented in the Map window and the Results panel. How do each of the three parameters affect the behavior of BULC in its results? Use the thumbnail to assess your subjective satisfaction with the results, and use the Inspector to view the BULC behavior in individual pixels. Can you produce an optimal outcome for this given set of input classifications?

    +

    BULC results can be viewed interactively, allowing you to view more detailed estimations of the LULC around the study area. We will zoom into a specific area where change did occur after 2016. To do that, turn on the Satellite view and zoom in. Watching the scale bar in the lower right of the Map panel, continue zooming until the scale bar says 5 km. Then, enter “-60.742, -9.844” in the Earth Engine search tool, located above the code. The text will be interpreted as a longitude/latitude value and will offer you a nearby coordinate, indicated with a value for the degrees West and the degrees South. Click that entry and Earth Engine will move to that location, while keeping at the specified zoom level. Let’s compare the BULC result in this sector against the image from Earth Engine’s satellite view that is underneath it (Fig. 4.8.4).

    +

    +
    +
    +

    +

    Fig. 4.8.4 Comparison of the final classification of the northern part of the study area to the satellite view

    +
    +
    +

    BULC captured the changes between 2016 and 2021 with a classification series that suggests agricultural development (Fig. 4.8.4, left). Given the appearance of BULC’s 2021 classification, it suggests that the satellite backdrop at the time of this writing (Fig. 4.8.4, right) came from an earlier time period.

    +

    Now, in the Results panel, select BULC, then Movie. Set your desired frame speed and resolution, then select Redraw Thumbnail. Then, zoom the main Map even closer to some agriculture that appears to have been established between 2016 and 2021. Redraw the thumbnail movie as needed to find an interesting set of pixels.

    +

    With this finer-scale access to the results of BULC, you can select individual pixels to inspect. Move the horizontal divider downward to expose the Inspector tab and Console tab. Use the Inspector to click on several pixels to learn their history as expressed in the inputted Events and in BULC’s interpretation of the noise and signal in the Event series. In a chosen pixel, you might see output that looks like Fig. 4.8.5. It indicates a possible conversion in the Event time series after a few classifications of the pixel as Forest. This decreases the confidence that the pixel is still Forest (Fig. 4.8.5, lower panel), but not enough for the Active Agriculture class (class 3) to become the dominant probability. After the subsequent Event labels the pixel as Forest, the confidence (lower panel) recovers slightly, but not to its former level. The next Event classifies the pixel as Active Agriculture, confidently, by interpreting that second Active Agriculture classification, in a setting where change was already somewhat suspected after the first non-Forest classification. BULC’s label (middle panel) changes to be Active Agriculture at that point in the sequence. Subsequent Event classifications as Active Agriculture creates a growing confidence that its proper label at the end of the sequence was indeed Active Agriculture.

    +

    Fig. 4.8.5 History for 2016-2020 for a pixel that appeared to have been newly cultivated during that period. (above): the input classifications, which sFig. 4.8.5 History for 2016-2020 for a pixel that appeared to have been newly cultivated during that period. (above): the input classifications, which sFig. 4.8.5 History for 2016-2020 for a pixel that appeared to have been newly cultivated during that period. (above): the input classifications, which s

    +

    Question 3. Run the code again with the same data, but adjust the three levelers, then view the results presented in the Map window and the Results panel. How do each of the three parameters affect the behavior of BULC in its results? Use the thumbnail to assess your subjective satisfaction with the results, and use the Inspector to view the BULC behavior in individual pixels. Can you produce an optimal outcome for this given set of input classifications?

    14.4 Change Detection with BULC-D

    -

    What if we wanted to identify areas of likely change or stability without trying to identify the initial and final LULC class? BULC-D is an algorithm that estimates, at each time step, the probability of noteworthy change. The example below uses the Normalized Burn Ratio (NBR) as a gauge: BULC-D assesses whether the ratio has meaningfully increased, decreased, or remained the same. It is then the choice of the analyst to decide how to treat these assessed probabilities of stability and change.

    +

    What if we wanted to identify areas of likely change or stability without trying to identify the initial and final LULC class? BULC-D is an algorithm that estimates, at each time step, the probability of noteworthy change. The example below uses the Normalized Burn Ratio (NBR) as a gauge: BULC-D assesses whether the ratio has meaningfully increased, decreased, or remained the same. It is then the choice of the analyst to decide how to treat these assessed probabilities of stability and change.

    BULC-D involves determining an expectation for an index across a user-specified time period and then comparing new values against that estimation. Using Bayesian logic, BULC-D then asks which of three hypotheses is most likely, given evidence from the new values to date from that index. The hypotheses are simple: Either the value has decreased meaningfully, or it has increased meaningfully, or it has not changed substantially compared to the previously established expectation. The details of the workings of BULC-D are beyond the scope of this exercise, but we provide it as a tool for exploration. BULC-D’s basic framework is the following:

    • Establish: Fit a harmonic curve with a user-specified number of terms to a stream of values from an index, such as the Normalized Difference Vegetation Index (NDVI), NBR, etc.
      • @@ -2556,10 +3005,10 @@ Note
      -

      Code Checkpoint F48c. The book’s repository contains information about accessing that interface.

      +

      Code Checkpoint F48c. The book’s repository contains information about accessing that interface.

      -

      After you have run the script to initialize the interface, BULC-D’s interface requires a few parameters to be set. For this run of BULC-D, we will set the parameters to the following:

      +

      After you have run the script to initialize the interface, BULC-D’s interface requires a few parameters to be set. For this run of BULC-D, we will set the parameters to the following:

      • Expectation years: 2020
      • Target year: 2021
        • @@ -2568,75 +3017,90 @@ Note
      • Harmonic fit: Yes, 1 harmonic term.

      Run BULC-D for this area. As a reminder, you should first zoom in enough that the scale bar reads “5 km” or finer. Then, search for the location “-60.7624, -9.8542”. When you run BULC-D, a result like Fig. F4.8.6 is shown for the layer of probabilities.

      -

       

      -

      Fig. 4.8.6 Result for BULC-D for the Roosevelt River area, depicting estimated probability of change and stability for 2021

      -

      The BULC-D image (Fig. F4.8.6) shows each pixel as a continuous three-value vector along a continuous range; the three values sum to 1. For example, a vector with values of [0.85, 0.10, 0.05] would represent an area estimated with high confidence according to BULC-D to have experienced a sustained drop in NBR in the target period compared to the values set by the expectation data. In that pixel, the combination of three colors would produce a value that is richly red. You can see Chap. F1.1 for more information on drawing bands of information to the screen using the red-green-blue additive color model in Earth Engine.

      -

      Each pixel experiences its own NBR history in both the expectation period and the target year. Next, we will highlight the history of three nearby areas: one, marked with a red balloon in your interface, that BULC assessed as having experienced a persistent drop in NBR; a second in green assessed to not have changed, and a third in blue assessed to have witnessed a persistent NBR increase. 

      -

      Figure F4.8.7 shows the NBR history for the red balloon in the southern part of the study area in Fig. F4.8.4. If you click on that pixel or one like it, you can see that, whereas the values were quite stable throughout the growing season for the years used to create the pixel’s expectation, they were persistently lower in the target year. This is flagged as a likely meaningful drop in the NBR by BULC-D, for consideration by the analyst.

      -

                                                                                             

      -

      Fig. 4.8.7 NBR history for a pixel with an apparent drop in NBR in the target year (below) as compared to the expectation years (above). Pixel is colored a shade of red in Fig. 4.8.6.

      -

      Figure F4.8.8 shows the NBR history for the blue balloon in the southern part of the study area in Fig. F4.8.4. For that pixel, while the values were quite stable throughout the growing season for the years used to create the pixel’s expectation, they were persistently higher in the target year. 

      -

      Question 4. Experiment with turning off one of the satellite sensor data sources used to create the expectation collection. For example, do you get the same results if the Sentinel-2 data stream is not used, or is the outcome different. You might make screen captures of the results to compare with Fig. 4.8.4. How strongly does each satellite stream affect the outcome of the estimate?  Do differences in the resulting estimate vary across the study area?

      -

      -

      Fig. 4.8.8 NBR history for a pixel with an apparent increase in NBR in the target year (below) as compared to the expectation years (above). Pixel is colored a shade of blue in Fig. 4.8.6.

      +

      +

      Fig. 4.8.6 Result for BULC-D for the Roosevelt River area, depicting estimated probability of change and stability for 2021

      +

      The BULC-D image (Fig. F4.8.6) shows each pixel as a continuous three-value vector along a continuous range; the three values sum to 1. For example, a vector with values of [0.85, 0.10, 0.05] would represent an area estimated with high confidence according to BULC-D to have experienced a sustained drop in NBR in the target period compared to the values set by the expectation data. In that pixel, the combination of three colors would produce a value that is richly red. You can see Chap. F1.1 for more information on drawing bands of information to the screen using the red-green-blue additive color model in Earth Engine.

      +

      Each pixel experiences its own NBR history in both the expectation period and the target year. Next, we will highlight the history of three nearby areas: one, marked with a red balloon in your interface, that BULC assessed as having experienced a persistent drop in NBR; a second in green assessed to not have changed, and a third in blue assessed to have witnessed a persistent NBR increase.

      +

      Figure F4.8.7 shows the NBR history for the red balloon in the southern part of the study area in Fig. F4.8.4. If you click on that pixel or one like it, you can see that, whereas the values were quite stable throughout the growing season for the years used to create the pixel’s expectation, they were persistently lower in the target year. This is flagged as a likely meaningful drop in the NBR by BULC-D, for consideration by the analyst.

      +

      +

      Fig. 4.8.7 NBR history for a pixel with an apparent drop in NBR in the target year (below) as compared to the expectation years (above). Pixel is colored a shade of red in Fig. 4.8.6.

      +

      Figure F4.8.8 shows the NBR history for the blue balloon in the southern part of the study area in Fig. F4.8.4. For that pixel, while the values were quite stable throughout the growing season for the years used to create the pixel’s expectation, they were persistently higher in the target year.

      +

      Question 4. Experiment with turning off one of the satellite sensor data sources used to create the expectation collection. For example, do you get the same results if the Sentinel-2 data stream is not used, or is the outcome different. You might make screen captures of the results to compare with Fig. 4.8.4. How strongly does each satellite stream affect the outcome of the estimate? Do differences in the resulting estimate vary across the study area?

      +
      +
      +

      +

      Fig. 4.8.8 NBR history for a pixel with an apparent increase in NBR in the target year (below) as compared to the expectation years (above). Pixel is colored a shade of blue in Fig. 4.8.6.

      +
      +

      Figure F4.8.8 also shows that, for that pixel, the fit of values for the years used to build the expectation showed a sine wave (shown in blue), but with a fit that was not very strong. When data for the target year was assembled (Fig. F4.8.8, bottom), the values were persistently above expectation throughout the growing season. Note that this pixel was identified as being different in the target year as compared to earlier years, which does not rule out the possibility that the LULC of the area was changed (for example, from Forest to Agriculture) during the years used to build the expectation collection. BULC-D is intended to be run steadily over a long period of time, with the changes marked as they occur, after which point the expectation would be recalculated.

      -

      -

      Fig. 4.8.9 NBR history for a pixel with no apparent increase or decrease in NBR in the target year (below) as compared to the expectation years (above). Pixel is colored a shade of green in Fig. 4.8.6.

      +
      +
      +

      +

      Fig. 4.8.9 NBR history for a pixel with no apparent increase or decrease in NBR in the target year (below) as compared to the expectation years (above). Pixel is colored a shade of green in Fig. 4.8.6.

      +
      +

      Fig. F4.8.9 shows the NBR history for the green balloon in the southern part of the study area in Fig. F4.8.4. For that pixel, the values in the expectation collection formed a sine wave, and the values in the target collection deviated only slightly from the expectation during the target year.

    14.5 Change Detection with BULC and Dynamic World

    -

    Recent advances in neural networks have made it easier to develop consistent models of LULC characteristics using satellite data. The Dynamic World project (Brown et al. 2022) applies a neural network, trained on a very large number of images, to each new Sentinel-2 image soon after it arrives. The result is a near-real-time classification interpreting the LULC of Earth’s surface, kept continually up to date with new imagery.

    +

    Recent advances in neural networks have made it easier to develop consistent models of LULC characteristics using satellite data. The Dynamic World project (Brown et al. 2022) applies a neural network, trained on a very large number of images, to each new Sentinel-2 image soon after it arrives. The result is a near-real-time classification interpreting the LULC of Earth’s surface, kept continually up to date with new imagery.

    What to do with the inevitable inconsistencies in a pixel’s stated LULC class through time? For a given pixel on a given image, its assigned class label is chosen by the Dynamic World algorithm as the maximum class probability given the band values on that day. Individual class probabilities are given as part of the dataset and could be used to better interpret a pixel’s condition and perhaps its history. Future work with BULC will involve incorporating these probabilities into BULC’s probability-based structure. For this tutorial, we will explore the consistency of the assigned labels in this same Roosevelt River area as a way to illustrate BULC’s potential for minimizing noise in this vast and growing dataset.

    14.5.1 5.1. Using BULC To Explore and Refine Dynamic World Classifications

    Code Checkpoint A48d. The book’s repository contains a script to use to begin this section. You will need to load the linked script and run it to begin.

    -

    After running the linked script, the BULC interface will initialize. Select Dynamic World from the dropdown menu where you earlier selected Image Collection. When you do, the interface opens several new fields to complete. BULC will need to know where you are interested in working with Dynamic World, since it could be anywhere on Earth. To specify the location, the interface field expects a nested list of lists of lists, which is modeled after the structure used inside the constructor ee.Geometry.Polygon. (When using drawing tools or specifying study areas using coordinates, you may have noticed this structure.) Enter the following nested list in the text field near the Dynamic World option, without enclosing it in quotes:

    +

    After running the linked script, the BULC interface will initialize. Select Dynamic World from the dropdown menu where you earlier selected Image Collection. When you do, the interface opens several new fields to complete. BULC will need to know where you are interested in working with Dynamic World, since it could be anywhere on Earth. To specify the location, the interface field expects a nested list of lists of lists, which is modeled after the structure used inside the constructor ee.Geometry.Polygon. (When using drawing tools or specifying study areas using coordinates, you may have noticed this structure.) Enter the following nested list in the text field near the Dynamic World option, without enclosing it in quotes:

    [[[-61.155, -10.559], [-60.285, -10.559], [-60.285, -9.436], [-61.155, -9.436]]]

    -

    Next, BULC will need to know which years of Dynamic World you are interested in. For this exercise, select 2021. Then, BULC will ask for the Julian days of the year that you are interested in. For this exercise, enter 150 for the start day and 300 for the end day. Because you selected Dynamic World for analysis in BULC, the interface defaults to offering the number 9 for the number of classes in Events and for the number of classes to track. This number represents the full set of classes in the Dynamic World classification scheme. You can leave other required settings shown in green with their default values. For the Color Output Palette, enter the following palette without quotes. This will render results in the Dynamic World default colors.

    +

    Next, BULC will need to know which years of Dynamic World you are interested in. For this exercise, select 2021. Then, BULC will ask for the Julian days of the year that you are interested in. For this exercise, enter 150 for the start day and 300 for the end day. Because you selected Dynamic World for analysis in BULC, the interface defaults to offering the number 9 for the number of classes in Events and for the number of classes to track. This number represents the full set of classes in the Dynamic World classification scheme. You can leave other required settings shown in green with their default values. For the Color Output Palette, enter the following palette without quotes. This will render results in the Dynamic World default colors.

    [‘419BDF’, ‘397D49’, ‘88B053’, ‘7A87C6’, ‘E49635’, ‘DFC35A’, ‘C4281B’, ‘A59B8F’, ‘B39FE1’]

    -

    When you have finished, select Apply Parameters at the bottom of the input panel. When it runs, BULC subsets the Dynamic World dataset to clip out according to the dates and location, identifying images from more than 40 distinct dates. The area covers two of the tiles in which Dynamic World classifications are partitioned to be served, so BULC receives more than 90 classifications. When BULC finishes its run, the Map panel will look like Fig. F4.8.10, BULC’s estimate of the final state of the landscape at the end of the classification sequence.

    -

    -

    Fig. F4.8.10 BULC classification using default settings for Roosevelt River area for late 2021

    -

    Let’s explore the suite of information returned by BULC about this time period in Dynamic World. Enter “Muiraquitã” in the search bar and view the results around that area to be able to see the changing LULC classifications within farm fields. Then, begin to inspect the results by viewing a Movie of the Events, with a data frame rate of 6 frames per second. Because the study area spans multiple Dynamic World tiles, you will find that many Event frames are black, meaning that there was no data in your sector on that particular image. Because of this, and also perhaps because of the very aggressive cloud masking built into Dynamic World, viewing Events (which, as a reminder, are the individual classified images directly from Dynamic World) can be a very challenging way to look for change and stability. BULC’s goal is to sift through those classifications to produce a time series that reflects, according to its estimation, the most likely LULC value at each time step. View the Movie of the BULC results and ask yourself whether each class is equally well replicated across the set of classifications. A still from midway through the Movie sequence of the BULC results can be seen in Fig. F4.8.11.

    -

    -

    Fig. F4.8.11 Still frame (right image) from the animation of BULC’s adjusted estimate of LULC through time near Muiraquitã

    +

    When you have finished, select Apply Parameters at the bottom of the input panel. When it runs, BULC subsets the Dynamic World dataset to clip out according to the dates and location, identifying images from more than 40 distinct dates. The area covers two of the tiles in which Dynamic World classifications are partitioned to be served, so BULC receives more than 90 classifications. When BULC finishes its run, the Map panel will look like Fig. F4.8.10, BULC’s estimate of the final state of the landscape at the end of the classification sequence.

    +

    Fig. F4.8.10 BULC classification using default settings for Roosevelt River area for late 2021Fig. F4.8.10 BULC classification using default settings for Roosevelt River area for late 2021

    +

    Let’s explore the suite of information returned by BULC about this time period in Dynamic World. Enter “Muiraquitã” in the search bar and view the results around that area to be able to see the changing LULC classifications within farm fields. Then, begin to inspect the results by viewing a Movie of the Events, with a data frame rate of 6 frames per second. Because the study area spans multiple Dynamic World tiles, you will find that many Event frames are black, meaning that there was no data in your sector on that particular image. Because of this, and also perhaps because of the very aggressive cloud masking built into Dynamic World, viewing Events (which, as a reminder, are the individual classified images directly from Dynamic World) can be a very challenging way to look for change and stability. BULC’s goal is to sift through those classifications to produce a time series that reflects, according to its estimation, the most likely LULC value at each time step. View the Movie of the BULC results and ask yourself whether each class is equally well replicated across the set of classifications. A still from midway through the Movie sequence of the BULC results can be seen in Fig. F4.8.11.

    +
    +
    +

    +

    Fig. F4.8.11 Still frame (right image) from the animation of BULC’s adjusted estimate of LULC through time near Muiraquitã

    +
    +

    As BULC uses the classification inputs to estimate the state of the LULC at each time step, it also tracks its confidence in those estimates. This is shown in several ways in the interface.

      -
    • You can view a Movie of BULC’s confidence through time as it reacts to the consistency or variability of the class identified in each pixel by Dynamic World. View that movie now over this area to see the evolution of BULC’s confidence through time of the class of each pixel. A still frame from this movie can be seen in Fig. F4.8.12. The frame and animation indicate that BULC’s confidence is lowest in pixels where the estimate flips between similar categories, such as Grass and Shrub & Scrub. It also is low at the edges of land covers, even where the covers (such as Forest and Water) are easy to discern from each other.  
      • -
    • You can inspect the final confidence estimate from BULC, which is shown as a grayscale image in the set of Map layers in the left lower panel. That single layer synthesizes how, across many Dynamic World classifications, the confidence in certain LULC classes and locations is ultimately more stable than in others. For example, generally speaking, the Forest class is classified consistently across this assemblage of Dynamic World images. Agriculture fields are less consistently classified as a single class, as evidenced by their relatively low confidence.
      • -
    • Another way of viewing BULC’s confidence is through the Inspector tab. You can click on individual pixels to view their values in the Event time series and in the BULC time series, and see BULC’s corresponding confidence value changing through time in response to the relative stability of each pixel’s classification.
      • +
    • You can view a Movie of BULC’s confidence through time as it reacts to the consistency or variability of the class identified in each pixel by Dynamic World. View that movie now over this area to see the evolution of BULC’s confidence through time of the class of each pixel. A still frame from this movie can be seen in Fig. F4.8.12. The frame and animation indicate that BULC’s confidence is lowest in pixels where the estimate flips between similar categories, such as Grass and Shrub & Scrub. It also is low at the edges of land covers, even where the covers (such as Forest and Water) are easy to discern from each other.
      +
      • +
    • You can inspect the final confidence estimate from BULC, which is shown as a grayscale image in the set of Map layers in the left lower panel. That single layer synthesizes how, across many Dynamic World classifications, the confidence in certain LULC classes and locations is ultimately more stable than in others. For example, generally speaking, the Forest class is classified consistently across this assemblage of Dynamic World images. Agriculture fields are less consistently classified as a single class, as evidenced by their relatively low confidence.
      • +
    • Another way of viewing BULC’s confidence is through the Inspector tab. You can click on individual pixels to view their values in the Event time series and in the BULC time series, and see BULC’s corresponding confidence value changing through time in response to the relative stability of each pixel’s classification.
    • Another way to view BULC’s confidence estimation is as a hillshade enhancement of the final BULC classification. If you select the Probability Hillshade in the set of Map layers, it shows the final BULC classification as a textured surface, in which you can see where lower-confidence pixels are classified.
    -

    -

    Fig. F4.8.12 Still frame from the animation of changing confidence through time, near Muiraquitã.

    +
    +
    +

    +

    Fig. F4.8.12 Still frame from the animation of changing confidence through time, near Muiraquitã.

    +
    +

    14.5.2 5.2. Using BULC To Visualize Uncertainty of Dynamic World in Simplified Categories

    -

    In the previous section, you may have noticed that there are two main types of uncertainty in BULC’s assessment of long-term classification confidence. One type is due to spatial uncertainty at the edge of two relatively distinct phenomena, like the River/Forest boundary visible in Fig. F4.8.12. These are shown in dark tones in the confidence images, and emphasized in the Probability Hillshade. The other type of uncertainty is due to some cause of labeling uncertainty, due either (1) to the similarity of the classes, or (2) to persistent difficulty in distinguishing two distinct classes that are meaningfully different but spectrally similar. An example of uncertainty due to similar labels is distinguishing flooded and non-flooded wetlands in classifications that contain both those categories. An example of difficulty distinguishing distinct but spectrally similar classes might be distinguishing a parking lot from a body of water.

    -

    BULC allows you to remap the classifications it is given as input, compressing categories as a way to minimize uncertainty due to similarity among classes. In the setting of Dynamic World in this study area, we notice that several classes are functionally similar for the purposes of detecting new deforestation: Farm fields and pastures are variously labeled on any given Dynamic World classification as Grass, Flooded Vegetation, Crops, Shrub & Scrub, Built, or Bare Ground. What if we wanted to combine these categories to be similar to the distinctions of the classified Events from this lab’s Sect. 1? The classes in that section were Forest, Water, and Active Agriculture. To remap the Dynamic World classification, continue with the same run as in Sect. 5.1. Near where you specified the location for clipping Dynamic World, there are two fields for remapping. Select the Remap checkbox and in the “from” field, enter (without quotes):

    +

    In the previous section, you may have noticed that there are two main types of uncertainty in BULC’s assessment of long-term classification confidence. One type is due to spatial uncertainty at the edge of two relatively distinct phenomena, like the River/Forest boundary visible in Fig. F4.8.12. These are shown in dark tones in the confidence images, and emphasized in the Probability Hillshade. The other type of uncertainty is due to some cause of labeling uncertainty, due either (1) to the similarity of the classes, or (2) to persistent difficulty in distinguishing two distinct classes that are meaningfully different but spectrally similar. An example of uncertainty due to similar labels is distinguishing flooded and non-flooded wetlands in classifications that contain both those categories. An example of difficulty distinguishing distinct but spectrally similar classes might be distinguishing a parking lot from a body of water.

    +

    BULC allows you to remap the classifications it is given as input, compressing categories as a way to minimize uncertainty due to similarity among classes. In the setting of Dynamic World in this study area, we notice that several classes are functionally similar for the purposes of detecting new deforestation: Farm fields and pastures are variously labeled on any given Dynamic World classification as Grass, Flooded Vegetation, Crops, Shrub & Scrub, Built, or Bare Ground. What if we wanted to combine these categories to be similar to the distinctions of the classified Events from this lab’s Sect. 1? The classes in that section were Forest, Water, and Active Agriculture. To remap the Dynamic World classification, continue with the same run as in Sect. 5.1. Near where you specified the location for clipping Dynamic World, there are two fields for remapping. Select the Remap checkbox and in the “from” field, enter (without quotes):

    0,1,2,3,4,5,6,7,8

    In the “to” field, enter (without quotes):

    1,0,2,2,2,2,2,2,0

    This directs BULC to create a three-class remap of each Dynamic World image. Next, in the area of the interface where you specify the palette, enter the same palette used earlier:

    [‘green’, ‘blue’, ‘yellow’]

    -

    Before continuing, think for a moment about how many classes you have now. From BULC’s perspective, the Dynamic World events will have 3 classes and you will be tracking 3 classes. Set both the Number of Classes in Events and Number of Classes to Track to 3. Then click Apply Parameters to send this new run to BULC.

    -

    The confidence image shown in the main Map panel is instructive (Fig. 4.8.13). Using data from 2020, 2021, and 2022, It indicates that much of the uncertainty among the original Dynamic World classifications was in distinguishing labels within agricultural fields. When that uncertainty is removed by combining classes, the BULC result indicates that a substantial part of the remaining uncertainty is at the edges of distinct covers. For example, in the south-central and southern part of the frame, much of the uncertainty among classifications in the original Dynamic World classifications was due to distinction among the highly similar, easily confused classes. Much of what remained (right) after remapping (right) formed outlines of the river and the edges between farmland and forest: a graphic depiction of the “spatial uncertainty” discussed earlier. Yet not all of the uncertainty was spatial; the thicker, darker areas of uncertainty even after remapping (right, at the extreme eastern edge for example) indicates a more fundamental disagreement in the classifications. In those pixels, even when the Agriculture-like classes were compressed, there was still considerable uncertainty (likely between Forest and Active Agriculture) in the true state of these areas. These might be of further interest: were they places newly deforested in 2020-2022? Were they abandoned fields regrowing?  Were they degraded at some point? The mapping of uncertainty may hold promise for a better understanding of uncertainty as it is encountered in real classifications, thanks to Dynamic World.

    -

    -

    Fig. F4.8.13 Final confidence layer from the run with (left) and without (right) remapping to combine similar LULC classes to distinguish Forest, Water, and Active Agriculture near -60.696W, -9.826S

    +

    Before continuing, think for a moment about how many classes you have now. From BULC’s perspective, the Dynamic World events will have 3 classes and you will be tracking 3 classes. Set both the Number of Classes in Events and Number of Classes to Track to 3. Then click Apply Parameters to send this new run to BULC.

    +

    The confidence image shown in the main Map panel is instructive (Fig. 4.8.13). Using data from 2020, 2021, and 2022, It indicates that much of the uncertainty among the original Dynamic World classifications was in distinguishing labels within agricultural fields. When that uncertainty is removed by combining classes, the BULC result indicates that a substantial part of the remaining uncertainty is at the edges of distinct covers. For example, in the south-central and southern part of the frame, much of the uncertainty among classifications in the original Dynamic World classifications was due to distinction among the highly similar, easily confused classes. Much of what remained (right) after remapping (right) formed outlines of the river and the edges between farmland and forest: a graphic depiction of the “spatial uncertainty” discussed earlier. Yet not all of the uncertainty was spatial; the thicker, darker areas of uncertainty even after remapping (right, at the extreme eastern edge for example) indicates a more fundamental disagreement in the classifications. In those pixels, even when the Agriculture-like classes were compressed, there was still considerable uncertainty (likely between Forest and Active Agriculture) in the true state of these areas. These might be of further interest: were they places newly deforested in 2020-2022? Were they abandoned fields regrowing? Were they degraded at some point? The mapping of uncertainty may hold promise for a better understanding of uncertainty as it is encountered in real classifications, thanks to Dynamic World.

    +

    Fig. F4.8.13 Final confidence layer from the run with (left) and without (right) remapping to combine similar LULC classes to distinguish Forest, Water, and Active Agriculture near -60.696W, -9.826SFig. F4.8.13 Final confidence layer from the run with (left) and without (right) remapping to combine similar LULC classes to distinguish Forest, Water, and Active Agriculture near -60.696W, -9.826S

    Given the tools and approaches presented in this lab, you should now be able to import your own classifications for BULC (Sects. 1–3), detect changes in sets of raw imagery (Sect. 4), or use Dynamic World’s pre-created classifications (Sect. 5). The following exercises explore this potential.

    Synthesis

    Assignment 1. For a given set of classifications as inputs, BULC uses three parameters that specify how strongly to trust the initial classification, how heavily to weigh the evidence of each classification, and how to adjust the confidence at the end of each time step. For this exercise, adjust the values of these three parameters to explore the strength of the effect they can have on the BULC results.

    -

    Assignment 2. The BULC-D framework produces a continuous three-value vector of the probability of change at each pixel. This variability accounts for the mottled look of the figures when those probabilities are viewed across space. Use the Inspector tool or the interface to explore the final estimated probabilities, both numerically and as represented by different colors of pixels in the given example. Compare and contrast the mean NBR values from the earlier and later years, which are drawn in the Layer list. Then answer the following questions:

    +

    Assignment 2. The BULC-D framework produces a continuous three-value vector of the probability of change at each pixel. This variability accounts for the mottled look of the figures when those probabilities are viewed across space. Use the Inspector tool or the interface to explore the final estimated probabilities, both numerically and as represented by different colors of pixels in the given example. Compare and contrast the mean NBR values from the earlier and later years, which are drawn in the Layer list. Then answer the following questions:

      -
    1. In general, how well does BULC-D appear to be identifying locations of likely change?
      2. +
    2. In general, how well does BULC-D appear to be identifying locations of likely change?
    3. Does one type of change (decrease, increase, no change) appear to be mapped better than the others? If so, why do you think this is?

    Assignment 3. The BULC-D example used here was for 2021. Run it for 2022 or later at this location. How well do results from adjacent years complement each other?

    -

    Assignment 4. Run BULC-D in a different area for a year of interest of your choosing. How do you like the results?

    +

    Assignment 4. Run BULC-D in a different area for a year of interest of your choosing. How do you like the results?

    Assignment 5. Describe how you might use BULC-D as a filter for distinguishing meaningful change from noise. In your answer, you can consider using BULC-D before or after BULC or some other time-series algorithm, like CCDC or LandTrendr.

    Assignment 6. Analyze stability and change with Dynamic World for other parts of the world and for other years. For example, you might consider:

      @@ -2645,16 +3109,16 @@ Note

      [[[-71.578, 49.755], [-71.578, 49.445], [-70.483, 49.445], [-70.483, 49.755]]]

      Location of a summer 2020 fire

        -

      1. Addis Ababa, Ethiopia:  [[[38.79, 9.00], [38.79, 8.99], [38.81, 8.99], [38.81, 9.00]]]

        2. +

      2. Addis Ababa, Ethiopia: [[[38.79, 9.00], [38.79, 8.99], [38.81, 8.99], [38.81, 9.00]]]

      3. Calacalí, Ecuador: [[[-78.537, 0.017], [-78.537, -0.047], [-78.463, -0.047], [-78.463, 0.017]]]

      4. Irpin, Ukraine: [[[30.22, 50.58], [30.22, 50.525], [30.346, 50.525], [30.346, 50.58]]]

        5. -

      5. A different location of your own choosing. To do this, use the Earth Engine drawing tools to draw a rectangle somewhere on Earth. Then, at the top of the Import section, you will see an icon that looks like a sheet of paper. Click that icon and look for the polygon specification for the rectangle you drew. Paste that into the location field for the Dynamic World interface.

        6. +

      6. A different location of your own choosing. To do this, use the Earth Engine drawing tools to draw a rectangle somewhere on Earth. Then, at the top of the Import section, you will see an icon that looks like a sheet of paper. Click that icon and look for the polygon specification for the rectangle you drew. Paste that into the location field for the Dynamic World interface.

    Conclusion

    In this lab, you have viewed several related but distinct ways to use Bayesian statistics to identify locations of LULC change in complex landscapes. While they are standalone algorithms, they are each intended to provide a perspective either on the likelihood of change (BULC-D) or of extracting signal from noisy classifications (BULC). You can consider using them especially when you have pixels that, despite your best efforts, periodically flip back and forth between similar but different classes. BULC can help ignore noise, and BULC-D can help reveal whether this year’s signal has precedent in past years.

    -

    To learn more about the BULC algorithm, you can view this interactive probability illustration tool by a link found in script F48s1 - Supplemental in the book’s repository. In the future, after you have learned how to use the logic of BULC, you might prefer to work with the JavaScript code version. To do that, you can find a tutorial at the website of the authors.

    +

    To learn more about the BULC algorithm, you can view this interactive probability illustration tool by a link found in script F48s1 - Supplemental in the book’s repository. In the future, after you have learned how to use the logic of BULC, you might prefer to work with the JavaScript code version. To do that, you can find a tutorial at the website of the authors.

    References

    @@ -2667,20 +3131,29 @@ Note
    -

    15 Exploring Lagged Effects in Time Series 

    -

    ::: {.callout-tip} # Chapter Information

    +

    15 Exploring Lagged Effects in Time Series

    +
    +
    +
    + +
    +
    +Chapter Information +
    +
    +

    Author

    Andréa Puzzi Nicolau, Karen Dyson, David Saah, Nicholas Clinton

    Overview

    -

    In this chapter, we will introduce lagged effects to build on previous work in modeling time-series data. Time-lagged effects occur when an event at one point in time impacts dependent variables at a later point in time. You will be introduced to concepts of autocovariance and autocorrelation, cross-covariance and cross-correlation, and auto-regressive models. At the end of this chapter, you will be able to examine how variables relate to one another across time, and to fit time series models that take into account lagged events. 

    +

    In this chapter, we will introduce lagged effects to build on previous work in modeling time-series data. Time-lagged effects occur when an event at one point in time impacts dependent variables at a later point in time. You will be introduced to concepts of autocovariance and autocorrelation, cross-covariance and cross-correlation, and auto-regressive models. At the end of this chapter, you will be able to examine how variables relate to one another across time, and to fit time series models that take into account lagged events.

    Learning Outcomes

      -
    • Using the ee.Join function to create time-lagged collections.
      • +
    • Using the ee.Join function to create time-lagged collections.
    • Calculating autocovariance and autocorrelation.
    • Calculating cross-covariance and cross-correlation.
    • Fitting auto-regressive models.
      • @@ -2691,137 +3164,160 @@ Note
      • Import images and image collections, filter, and visualize (Part F1).
      • Perform basic image analysis: select bands, compute indices, create masks, classify images (Part F2).
        • -
      • ​​Create a graph using ui.Chart (Chap. F1.3).
        • -
      • Write a function and map it over an ImageCollection (Chap. F4.0).
        • +
      • ​​Create a graph using ui.Chart (Chap. F1.3).
        • +
      • Write a function and map it over an ImageCollection (Chap. F4.0).
      • Mask cloud, cloud shadow, snow/ice, and other undesired pixels (Chap. F4.3).
        • -
      • Fit linear and nonlinear functions with regression in an ImageCollection time series (Chap. F4.6).
        • +
      • Fit linear and nonlinear functions with regression in an ImageCollection time series (Chap. F4.6).
    +
    +

    Introduction

    While fitting functions to time series allows you to account for seasonality in your models, sometimes the impact of a seasonal event does not impact your dependent variable until the next month, the next year, or even multiple years later. For example, coconuts take 18–24 months to develop from flower to harvestable size. Heavy rains during the flower development stage can severely reduce the number of coconuts that can be harvested months later, with significant negative economic repercussions. These patterns—where events in one time period impact our variable of interest in later time periods—are important to be able to include in our models.

    In this chapter, we introduce lagged effects into our previous discussions on interpreting time-series data (Chaps. F4.6 and F4.7). Being able to integrate lagged effects into our time-series models allows us to address many important questions. For example, streamflow can be accurately modeled by taking into account previous streamflow, rainfall, and soil moisture; this improved understanding helps predict and mitigate the impacts of drought and flood events made more likely by climate change (Sazib et al. 2020). As another example, time-series lag analysis was able to determine that decreased rainfall was associated with increases in livestock disease outbreaks one year later in India (Karthikeyan et al. 2021).

    -

    15.1 Autocovariance and Autocorrelation

    -

    Before we dive into autocovariance and autocorrelation, let’s set up an area of interest and dataset that we can use to illustrate these concepts. We will work with a detrended time series (as seen in Chap. F4.6) based on the USGS Landsat 8 Level 2, Collection 2, Tier 1 image collection. Copy and paste the code below to filter the Landsat 8 collection to a point of interest over California and specific dates, and apply the pre-processing function—to mask clouds (as seen in Chap. F4.3) and to scale and add variables of interest (as seen in Chap. F4.6).

    -

    // Define function to mask clouds, scale, and add variables
    -// (NDVI, time and a constant) to Landsat 8 imagery.
    -function maskScaleAndAddVariable(image) {   // Bit 0 - Fill   // Bit 1 - Dilated Cloud   // Bit 2 - Cirrus   // Bit 3 - Cloud   // Bit 4 - Cloud Shadow   var qaMask = image.select(‘QA_PIXEL’).bitwiseAnd(parseInt(‘11111’,       2)).eq(0);   var saturationMask = image.select(‘QA_RADSAT’).eq(0);   // Apply the scaling factors to the appropriate bands.   var opticalBands = image.select(‘SR_B.’).multiply(0.0000275).add(-       0.2);   var thermalBands = image.select(‘ST_B.’).multiply(0.00341802)
    -       .add(149.0);   // Replace the original bands with the scaled ones and apply the masks.   var img = image.addBands(opticalBands, null, true)
    -       .addBands(thermalBands, null, true)
    -       .updateMask(qaMask)
    -       .updateMask(saturationMask);   var imgScaled = image.addBands(img, null, true);   // Now we start to add variables of interest.   // Compute time in fractional years since the epoch.   var date = ee.Date(image.get(’system:time_start’));   var years = date.difference(ee.Date(’1970-01-01’), ’year’);   var timeRadians = ee.Image(years.multiply(2      Math.PI));
    -   // Return the image with the added bands.
    -   return imgScaled
    -       // Add an NDVI band.
    -       .addBands(imgScaled.normalizedDifference([’SR_B5’, ’SR_B4’])
    -           .rename(’NDVI’))       // Add a time band.       .addBands(timeRadians.rename(‘t’))
    -       .float()       // Add a constant band.       .addBands(ee.Image.constant(1));
    -}

    -

    // Import region of interest. Area over California.
    -var roi = ee.Geometry.Polygon([
    -   [-119.44617458417066,35.92639730653253],
    -   [-119.07675930096754,35.92639730653253],
    -   [-119.07675930096754,36.201704711823844],
    -   [-119.44617458417066,36.201704711823844],
    -   [-119.44617458417066,35.92639730653253]
    -]);

    -

    // Import the USGS Landsat 8 Level 2, Collection 2, Tier 1 collection,
    -// filter, mask clouds, scale, and add variables.
    -var landsat8sr = ee.ImageCollection(‘LANDSAT/LC08/C02/T1_L2’)
    -   .filterBounds(roi)
    -   .filterDate(‘2013-01-01’, ‘2018-01-01’)
    -   .map(maskScaleAndAddVariable);

    -

    // Set map center.
    -Map.centerObject(roi, 10);

    -

    Next, copy and paste the code below to estimate the linear trend using the linearRegression reducer, and remove that linear trend from the time series.

    -

    // List of the independent variable names.
    -var independents = ee.List([‘constant’, ‘t’]);

    -

    // Name of the dependent variable.
    -var dependent = ee.String(‘NDVI’);

    -

    // Compute a linear trend.  This will have two bands: ‘residuals’ and
    -// a 2x1 band called coefficients (columns are for dependent variables).
    -var trend = landsat8sr.select(independents.add(dependent))
    -   .reduce(ee.Reducer.linearRegression(independents.length(), 1));

    -

    // Flatten the coefficients into a 2-band image
    -var coefficients = trend.select(‘coefficients’)   // Get rid of extra dimensions and convert back to a regular image   .arrayProject([0])
    -   .arrayFlatten([independents]);

    -

    // Compute a detrended series.
    -var detrended = landsat8sr.map(function(image) {   return image.select(dependent)
    -       .subtract(image.select(independents).multiply(
    -               coefficients)
    -           .reduce(‘sum’))
    -       .rename(dependent)
    -       .copyProperties(image, [‘system:time_start’]);
    -});

    -

    Now let’s turn to autocovariance and autocorrelation. The autocovariance of a time series refers to the dependence of values in the time series at time t with values at time h = t − lag. The autocorrelation is the correlation between elements of a dataset at one time and elements of the same dataset at a different time. The autocorrelation is the autocovariance normalized by the standard deviations of the covariates. Specifically, we assume our time series is stationary, and define the autocovariance and autocorrelation following Shumway and Stoffer (2019). Comparing values at time t to previous values is useful not only for computing autocovariance, but also for a variety of other time series analyses as you’ll see shortly.

    -

    To combine image data with previous values in Earth Engine, the first step is to join the previous values to the current values. To do that, we will use a ee.Join function to create what we’ll call a lagged collection. Copy and paste the code below to define a function that creates a lagged collection.

    -

    // Function that creates a lagged collection.
    -var lag = function(leftCollection, rightCollection, lagDays) {   var filter = ee.Filter.and(       ee.Filter.maxDifference({
    -           difference: 1000 * 60 * 60 * 24 * lagDays,
    -           leftField: ‘system:time_start’,
    -           rightField: ‘system:time_start’       }),       ee.Filter.greaterThan({
    -           leftField: ‘system:time_start’,
    -           rightField: ‘system:time_start’       }));   return ee.Join.saveAll({
    -       matchesKey: ‘images’,
    -       measureKey: ‘delta_t’,
    -       ordering: ‘system:time_start’,
    -       ascending: false, // Sort reverse chronologically   }).apply({
    -       primary: leftCollection,
    -       secondary: rightCollection,
    -       condition: filter
    -   });
    -};

    -

    This function joins a collection to itself, using a filter that gets all the images before each image’s date that are within a specified time difference (in days) of each image. That list of previous images within the lag time is stored in a property of the image called images, sorted reverse chronologically. For example, to create a lagged collection from the detrended Landsat imagery, copy and paste:

    -

    // Create a lagged collection of the detrended imagery.
    -var lagged17 = lag(detrended, detrended, 17);

    +

    15.1 Autocovariance and Autocorrelation

    +

    Before we dive into autocovariance and autocorrelation, let’s set up an area of interest and dataset that we can use to illustrate these concepts. We will work with a detrended time series (as seen in Chap. F4.6) based on the USGS Landsat 8 Level 2, Collection 2, Tier 1 image collection. Copy and paste the code below to filter the Landsat 8 collection to a point of interest over California and specific dates, and apply the pre-processing function—to mask clouds (as seen in Chap. F4.3) and to scale and add variables of interest (as seen in Chap. F4.6).

    +
    // Define function to mask clouds, scale, and add variables  
+// (NDVI, time and a constant) to Landsat 8 imagery.  
+function maskScaleAndAddVariable(image) {   // Bit 0 - Fill   // Bit 1 - Dilated Cloud   // Bit 2 - Cirrus   // Bit 3 - Cloud   // Bit 4 - Cloud Shadow   var qaMask = image.select('QA_PIXEL').bitwiseAnd(parseInt('11111',       2)).eq(0);   var saturationMask = image.select('QA_RADSAT').eq(0);   // Apply the scaling factors to the appropriate bands.   var opticalBands = image.select('SR_B.').multiply(0.0000275).add(-       0.2);   var thermalBands = image.select('ST_B.*').multiply(0.00341802)  
+       .add(149.0);   // Replace the original bands with the scaled ones and apply the masks.   var img = image.addBands(opticalBands, null, true)  
+       .addBands(thermalBands, null, true)  
+       .updateMask(qaMask)  
+       .updateMask(saturationMask);   var imgScaled = image.addBands(img, null, true);   // Now we start to add variables of interest.   // Compute time in fractional years since the epoch.   var date = ee.Date(image.get('system:time_start'));   var years = date.difference(ee.Date('1970-01-01'), 'year');   var timeRadians = ee.Image(years.multiply(2 * Math.PI));  
+   // Return the image with the added bands.  
+   return imgScaled  
+       // Add an NDVI band.  
+       .addBands(imgScaled.normalizedDifference(['SR_B5', 'SR_B4'])  
+           .rename('NDVI'))       // Add a time band.       .addBands(timeRadians.rename('t'))  
+       .float()       // Add a constant band.       .addBands(ee.Image.constant(1));  
+}  
+  
+// Import region of interest. Area over California.  
+var roi = ee.Geometry.Polygon([  
+   [-119.44617458417066,35.92639730653253],  
+   [-119.07675930096754,35.92639730653253],  
+   [-119.07675930096754,36.201704711823844],  
+   [-119.44617458417066,36.201704711823844],  
+   [-119.44617458417066,35.92639730653253]  
+]);  
+  
+  
+// Import the USGS Landsat 8 Level 2, Collection 2, Tier 1 collection,  
+// filter, mask clouds, scale, and add variables.  
+var landsat8sr = ee.ImageCollection('LANDSAT/LC08/C02/T1_L2')  
+   .filterBounds(roi)  
+   .filterDate('2013-01-01', '2018-01-01')  
+   .map(maskScaleAndAddVariable);  
+  
+// Set map center.  
+Map.centerObject(roi, 10);
    +

    Next, copy and paste the code below to estimate the linear trend using the linearRegression reducer, and remove that linear trend from the time series.

    +
    // List of the independent variable names.  
+var independents = ee.List(['constant', 't']);  
+  
+// Name of the dependent variable.  
+var dependent = ee.String('NDVI');  
+  
+// Compute a linear trend.  This will have two bands: 'residuals' and  
+// a 2x1 band called coefficients (columns are for dependent variables).  
+var trend = landsat8sr.select(independents.add(dependent))  
+   .reduce(ee.Reducer.linearRegression(independents.length(), 1));  
+  
+// Flatten the coefficients into a 2-band image  
+var coefficients = trend.select('coefficients')   // Get rid of extra dimensions and convert back to a regular image   .arrayProject([0])  
+   .arrayFlatten([independents]);  
+  
+// Compute a detrended series.  
+var detrended = landsat8sr.map(function(image) {   return image.select(dependent)  
+       .subtract(image.select(independents).multiply(  
+               coefficients)  
+           .reduce('sum'))  
+       .rename(dependent)  
+       .copyProperties(image, ['system:time_start']);  
+});
    +

    Now let’s turn to autocovariance and autocorrelation. The autocovariance of a time series refers to the dependence of values in the time series at time t with values at time h = t − lag. The autocorrelation is the correlation between elements of a dataset at one time and elements of the same dataset at a different time. The autocorrelation is the autocovariance normalized by the standard deviations of the covariates. Specifically, we assume our time series is stationary, and define the autocovariance and autocorrelation following Shumway and Stoffer (2019). Comparing values at time t to previous values is useful not only for computing autocovariance, but also for a variety of other time series analyses as you’ll see shortly.

    +

    To combine image data with previous values in Earth Engine, the first step is to join the previous values to the current values. To do that, we will use a ee.Join function to create what we’ll call a lagged collection. Copy and paste the code below to define a function that creates a lagged collection.

    +
    // Function that creates a lagged collection.  
+var lag = function(leftCollection, rightCollection, lagDays) {   var filter = ee.Filter.and(       ee.Filter.maxDifference({  
+           difference: 1000 * 60 * 60 * 24 * lagDays,  
+           leftField: 'system:time_start',  
+           rightField: 'system:time_start'       }),       ee.Filter.greaterThan({  
+           leftField: 'system:time_start',  
+           rightField: 'system:time_start'       }));   return ee.Join.saveAll({  
+       matchesKey: 'images',  
+       measureKey: 'delta_t',  
+       ordering: 'system:time_start',  
+       ascending: false, // Sort reverse chronologically   }).apply({  
+       primary: leftCollection,  
+       secondary: rightCollection,  
+       condition: filter  
+   });  
+};
    +

    This function joins a collection to itself, using a filter that gets all the images before each image’s date that are within a specified time difference (in days) of each image. That list of previous images within the lag time is stored in a property of the image called images, sorted reverse chronologically. For example, to create a lagged collection from the detrended Landsat imagery, copy and paste:

    +
    // Create a lagged collection of the detrended imagery.  
+var lagged17 = lag(detrended, detrended, 17);

    Why 17 days? Recall that the temporal cadence of Landsat is 16 days. Specifying 17 days in the join gets one previous image, but no more.

    -

    Now, we will compute the autocovariance using a reducer that expects a set of one-dimensional arrays as input. So pixel values corresponding to time t need to be stacked with pixel values at time t − lag as multiple bands in the same image. Copy and paste the code below to define a function to do so, and apply it to merge the bands from the lagged collection.

    -

    // Function to stack bands.
    -var merge = function(image) {   // Function to be passed to iterate.   var merger = function(current, previous) {       return ee.Image(previous).addBands(current);
    -   };   return ee.ImageCollection.fromImages(image.get(‘images’))
    -       .iterate(merger, image);
    -};

    -

    // Apply merge function to the lagged collection.
    -var merged17 = ee.ImageCollection(lagged17.map(merge));

    -

    Now the bands from time t and h are all in the same image. Note that the band name of a pixel at time h, ph, was the same as time t, pt (band name is “NDVI” in this case). During the merging process, it gets a ’_1’ appended to it (e.g. NDVI_1).

    -

    You can print the image collection to check the band names of one of the images. Copy and paste the code below to map a function to convert the merged bands to arrays with bands pt and ph, and then reduce it with the covariance reducer. We use a parallelScale factor of 8 in the reduce function to avoid the computation to run out of memory (this is not always needed). Note that the output of the covariance reducer is an array image, in which each pixel stores a 2x2 variance-covariance array. The off-diagonal elements are covariance, which you can map directly using the arrayGet function.

    -

    // Function to compute covariance.
    -var covariance = function(mergedCollection, band, lagBand) {   return mergedCollection.select([band, lagBand]).map(function(
    -       image) {       return image.toArray();
    -   }).reduce(ee.Reducer.covariance(), 8);
    -};

    -

    // Concatenate the suffix to the NDVI band.
    -var lagBand = dependent.cat(’_1’);

    -

    // Compute covariance.
    -var covariance17 = ee.Image(covariance(merged17, dependent, lagBand))
    -   .clip(roi);

    -

    // The output of the covariance reducer is an array image,
    -// in which each pixel stores a 2x2 variance-covariance array.
    -// The off diagonal elements are covariance, which you can map
    -// directly using:
    -Map.addLayer(covariance17.arrayGet([0, 1]),
    -   {
    -       min: 0,
    -       max: 0.02   },   ‘covariance (lag = 17 days)’);

    -

    Inspect the pixel values of the resulting covariance image (Fig. F4.9.1). The covariance is positive when the greater values of one variable (at time t) mainly correspond to the greater values of the other variable (at time h), and the same holds for the lesser values, therefore, the values tend to show similar behavior. In the opposite case, when the greater values of a variable correspond to the lesser values of the other variable, the covariance is negative.

    -

    -

    Fig. F4.9.1 Autocovariance image

    +

    Now, we will compute the autocovariance using a reducer that expects a set of one-dimensional arrays as input. So pixel values corresponding to time t need to be stacked with pixel values at time t − lag as multiple bands in the same image. Copy and paste the code below to define a function to do so, and apply it to merge the bands from the lagged collection.

    +
    // Function to stack bands.  
+var merge = function(image) {   // Function to be passed to iterate.   var merger = function(current, previous) {       return ee.Image(previous).addBands(current);  
+   };   return ee.ImageCollection.fromImages(image.get('images'))  
+       .iterate(merger, image);  
+};  
+  
+// Apply merge function to the lagged collection.  
+var merged17 = ee.ImageCollection(lagged17.map(merge));
    +

    Now the bands from time t and h are all in the same image. Note that the band name of a pixel at time h, ph, was the same as time t, pt (band name is “NDVI” in this case). During the merging process, it gets a ’_1’ appended to it (e.g. NDVI_1).

    +

    You can print the image collection to check the band names of one of the images. Copy and paste the code below to map a function to convert the merged bands to arrays with bands pt and ph, and then reduce it with the covariance reducer. We use a parallelScale factor of 8 in the reduce function to avoid the computation to run out of memory (this is not always needed). Note that the output of the covariance reducer is an array image, in which each pixel stores a 2x2 variance-covariance array. The off-diagonal elements are covariance, which you can map directly using the arrayGet function.

    +
    // Function to compute covariance.  
+var covariance = function(mergedCollection, band, lagBand) {   return mergedCollection.select([band, lagBand]).map(function(  
+       image) {       return image.toArray();  
+   }).reduce(ee.Reducer.covariance(), 8);  
+};  
+  
+// Concatenate the suffix to the NDVI band.  
+var lagBand = dependent.cat('_1');  
+  
+// Compute covariance.  
+var covariance17 = ee.Image(covariance(merged17, dependent, lagBand))  
+   .clip(roi);  
+  
+// The output of the covariance reducer is an array image,  
+// in which each pixel stores a 2x2 variance-covariance array.  
+// The off diagonal elements are covariance, which you can map  
+// directly using:  
+Map.addLayer(covariance17.arrayGet([0, 1]),  
+   {  
+       min: 0,  
+       max: 0.02   },   'covariance (lag = 17 days)');
    +

    Inspect the pixel values of the resulting covariance image (Fig. F4.9.1). The covariance is positive when the greater values of one variable (at time t) mainly correspond to the greater values of the other variable (at time h), and the same holds for the lesser values, therefore, the values tend to show similar behavior. In the opposite case, when the greater values of a variable correspond to the lesser values of the other variable, the covariance is negative.

    +
    +
    +

    +

    Fig. F4.9.1 Autocovariance image

    +
    +

    The diagonal elements of the variance-covariance array are variances. Copy and paste the code below to define and map a function to compute correlation (Fig. F4.9.2) from the variance-covariance array.

    -

    // Define the correlation function.
    -var correlation = function(vcArrayImage) {   var covariance = ee.Image(vcArrayImage).arrayGet([0, 1]);   var sd0 = ee.Image(vcArrayImage).arrayGet([0, 0]).sqrt();   var sd1 = ee.Image(vcArrayImage).arrayGet([1, 1]).sqrt();   return covariance.divide(sd0).divide(sd1).rename(       ‘correlation’);
    -};

    -

    // Apply the correlation function.
    -var correlation17 = correlation(covariance17).clip(roi);
    -Map.addLayer(correlation17,
    -   {
    -       min: -1,
    -       max: 1   },   ‘correlation (lag = 17 days)’);

    -

    -

    Fig. F4.9.2 Autocorrelation image

    +
    // Define the correlation function.  
+var correlation = function(vcArrayImage) {   var covariance = ee.Image(vcArrayImage).arrayGet([0, 1]);   var sd0 = ee.Image(vcArrayImage).arrayGet([0, 0]).sqrt();   var sd1 = ee.Image(vcArrayImage).arrayGet([1, 1]).sqrt();   return covariance.divide(sd0).divide(sd1).rename(       'correlation');  
+};  
+  
+// Apply the correlation function.  
+var correlation17 = correlation(covariance17).clip(roi);  
+Map.addLayer(correlation17,  
+   {  
+       min: -1,  
+       max: 1   },   'correlation (lag = 17 days)');
    +
    +
    +

    +

    Fig. F4.9.2 Autocorrelation image

    +
    +

    Higher positive values indicate higher correlation between the elements of the dataset, and lower negative values indicate the opposite.

    -

    It’s worth noting that you can do this for longer lags as well. Of course, that images list will fill up with all the images that are within lag of t. Those other images are also useful—for example, in fitting autoregressive models as described later.

    +

    It’s worth noting that you can do this for longer lags as well. Of course, that images list will fill up with all the images that are within lag of t. Those other images are also useful—for example, in fitting autoregressive models as described later.

    @@ -2832,52 +3328,67 @@ Note
    -

    Code Checkpoint F49a. The book’s repository contains a script that shows what your code should look like at this point.

    +

    Code Checkpoint F49a. The book’s repository contains a script that shows what your code should look like at this point.

    15.2 Cross-Covariance and Cross-Correlation

    -

    Cross-covariance is analogous to autocovariance, except instead of measuring the correspondence between a variable and itself at a lag, it measures the correspondence between a variable and a covariate at a lag. Specifically, we will define the cross-covariance and cross-correlation according to Shumway and Stoffer (2019). 

    +

    Cross-covariance is analogous to autocovariance, except instead of measuring the correspondence between a variable and itself at a lag, it measures the correspondence between a variable and a covariate at a lag. Specifically, we will define the cross-covariance and cross-correlation according to Shumway and Stoffer (2019).

    You already have all the code needed to compute cross-covariance and cross-correlation. But you do need a time series of another variable. Suppose we postulate that NDVI is related in some way to the precipitation before the NDVI was observed. To estimate the strength of this relationship in every pixel, copy and paste the code below to the existing script to load precipitation, join, merge, and reduce as previously:

    -

    // Precipitation (covariate)
    -var chirps = ee.ImageCollection(‘UCSB-CHG/CHIRPS/PENTAD’);

    -

    // Join the t-l (l=1 pentad) precipitation images to the Landsat.
    -var lag1PrecipNDVI = lag(landsat8sr, chirps, 5);

    -

    // Add the precipitation images as bands.
    -var merged1PrecipNDVI = ee.ImageCollection(lag1PrecipNDVI.map(merge));

    -

    // Compute and display cross-covariance.
    -var cov1PrecipNDVI = covariance(merged1PrecipNDVI, ‘NDVI’,   ‘precipitation’).clip(roi);
    -Map.addLayer(cov1PrecipNDVI.arrayGet([0, 1]), {},   ‘NDVI - PRECIP cov (lag = 5)’);

    -

    // Compute and display cross-correlation.
    -var corr1PrecipNDVI = correlation(cov1PrecipNDVI).clip(roi);
    -Map.addLayer(corr1PrecipNDVI, {
    -   min: -0.5,
    -   max: 0.5}, ‘NDVI - PRECIP corr (lag = 5)’);

    -

    What do you observe from this result? Looking at the cross-correlation image (Fig. F4.9.3), do you observe high values where you would expect high NDVI values (vegetated areas)? One possible drawback of this computation is that it’s only based on five days of precipitation, whichever five days came right before the NDVI image.

    -

    -

    Fig. F4.9.3 Cross-correlation image of NDVI and precipitation with a five-day lag.

    +
    // Precipitation (covariate)  
+var chirps = ee.ImageCollection('UCSB-CHG/CHIRPS/PENTAD');  
+  
+// Join the t-l (l=1 pentad) precipitation images to the Landsat.  
+var lag1PrecipNDVI = lag(landsat8sr, chirps, 5);  
+  
+// Add the precipitation images as bands.  
+var merged1PrecipNDVI = ee.ImageCollection(lag1PrecipNDVI.map(merge));  
+  
+// Compute and display cross-covariance.  
+var cov1PrecipNDVI = covariance(merged1PrecipNDVI, 'NDVI',   'precipitation').clip(roi);  
+Map.addLayer(cov1PrecipNDVI.arrayGet([0, 1]), {},   'NDVI - PRECIP cov (lag = 5)');  
+  
+// Compute and display cross-correlation.  
+var corr1PrecipNDVI = correlation(cov1PrecipNDVI).clip(roi);  
+Map.addLayer(corr1PrecipNDVI, {  
+   min: -0.5,  
+   max: 0.5}, 'NDVI - PRECIP corr (lag = 5)');
    +

    What do you observe from this result? Looking at the cross-correlation image (Fig. F4.9.3), do you observe high values where you would expect high NDVI values (vegetated areas)? One possible drawback of this computation is that it’s only based on five days of precipitation, whichever five days came right before the NDVI image.

    +
    +
    +

    +

    Fig. F4.9.3 Cross-correlation image of NDVI and precipitation with a five-day lag.

    +
    +

    Perhaps precipitation in the month before the observed NDVI is relevant? Copy and paste the code below to test the 30-day lag idea.

    -

    // Join the precipitation images from the previous month.
    -var lag30PrecipNDVI = lag(landsat8sr, chirps, 30);

    -

    var sum30PrecipNDVI = ee.ImageCollection(lag30PrecipNDVI.map(function(
    -   image) {   var laggedImages = ee.ImageCollection.fromImages(image
    -       .get(‘images’));   return ee.Image(image).addBands(laggedImages.sum()
    -       .rename(‘sum’));
    -}));

    -

    // Compute covariance.
    -var cov30PrecipNDVI = covariance(sum30PrecipNDVI, ‘NDVI’, ‘sum’).clip(
    -   roi);
    -Map.addLayer(cov1PrecipNDVI.arrayGet([0, 1]), {},   ‘NDVI - sum cov (lag = 30)’);

    -

    // Correlation.
    -var corr30PrecipNDVI = correlation(cov30PrecipNDVI).clip(roi);
    -Map.addLayer(corr30PrecipNDVI, {
    -   min: -0.5,
    -   max: 0.5}, ‘NDVI - sum corr (lag = 30)’);

    -

    Observe that the only change is to the merge method. Instead of merging the bands of the NDVI image and the covariate (precipitation), the entire list of precipitation is summed and added as a band (eliminating the need for iterate).

    -

    Which changes do you notice between the cross-correlation images—5 days lag vs. 30 days lag (Fig. F4.9.4)?. You can use the Inspector tool to assess if the correlation increased or not at vegetated areas.

    -

    -

    Fig. F4.9.4 Cross-correlation image of NDVI and precipitation with a 30-day lag.

    +
    // Join the precipitation images from the previous month.  
+var lag30PrecipNDVI = lag(landsat8sr, chirps, 30);  
+  
+var sum30PrecipNDVI = ee.ImageCollection(lag30PrecipNDVI.map(function(  
+   image) {   var laggedImages = ee.ImageCollection.fromImages(image  
+       .get('images'));   return ee.Image(image).addBands(laggedImages.sum()  
+       .rename('sum'));  
+}));  
+  
+// Compute covariance.  
+var cov30PrecipNDVI = covariance(sum30PrecipNDVI, 'NDVI', 'sum').clip(  
+   roi);  
+Map.addLayer(cov1PrecipNDVI.arrayGet([0, 1]), {},   'NDVI - sum cov (lag = 30)');  
+  
+// Correlation.  
+var corr30PrecipNDVI = correlation(cov30PrecipNDVI).clip(roi);  
+Map.addLayer(corr30PrecipNDVI, {  
+   min: -0.5,  
+   max: 0.5}, 'NDVI - sum corr (lag = 30)');
    +

    Observe that the only change is to the merge method. Instead of merging the bands of the NDVI image and the covariate (precipitation), the entire list of precipitation is summed and added as a band (eliminating the need for iterate).

    +

    Which changes do you notice between the cross-correlation images—5 days lag vs. 30 days lag (Fig. F4.9.4)?. You can use the Inspector tool to assess if the correlation increased or not at vegetated areas.

    +
    +
    +

    +

    Fig. F4.9.4 Cross-correlation image of NDVI and precipitation with a 30-day lag.

    +
    +

    As long as there is sufficient temporal overlap between the time series, these techniques could be extended to longer lags and longer time series.

    @@ -2889,55 +3400,60 @@ Note
    -

    Code Checkpoint F49b. The book’s repository contains a script that shows what your code should look like at this point.

    +

    Code Checkpoint F49b. The book’s repository contains a script that shows what your code should look like at this point.

    15.3 Auto-Regressive Models

    -

    The discussion of autocovariance preceded this section in order to introduce the concept of lag. Now that you have a way to get previous values of a variable, it’s worth considering auto-regressive models. Suppose that pixel values at time t depend in some way on previous pixel values—auto-regressive models are time series models that use observations from previous time steps as input to a regression equation to predict the value at the next time step. If you have observed significant, non-zero autocorrelations in a time series, this is a good assumption. Specifically, you may postulate a linear model such as the following, where pt is a pixel at time t, and et is a random error (Chap. F4.6):

    -

    pt = β0 + β1pt-1 + β2pt-2 + et                                        (F4.9.1)

    -

    To fit this model, you need a lagged collection as created previously, except with a longer lag (e.g., lag = 34 days). The next steps are to merge the bands, then reduce with the linear regression reducer.

    -

    Copy and paste the line below to the existing script to create a lagged collection, where the images list stores the two previous images:

    -

    var lagged34 = ee.ImageCollection(lag(landsat8sr, landsat8sr, 34));

    -

    Copy and paste the code below to merge the bands of the lagged collection such that each image has bands at time t and bands at times t - 1,…, t − lag. Note that it’s necessary to filter out any images that don’t have two previous temporal neighbors.

    -

    var merged34 = lagged34.map(merge).map(function(image) {   return image.set(‘n’, ee.List(image.get(‘images’))
    -       .length());
    +

    The discussion of autocovariance preceded this section in order to introduce the concept of lag. Now that you have a way to get previous values of a variable, it’s worth considering auto-regressive models. Suppose that pixel values at time t depend in some way on previous pixel values—auto-regressive models are time series models that use observations from previous time steps as input to a regression equation to predict the value at the next time step. If you have observed significant, non-zero autocorrelations in a time series, this is a good assumption. Specifically, you may postulate a linear model such as the following, where pt is a pixel at time t, and et is a random error (Chap. F4.6):

    +

    pt = β0 + β1pt-1 + β2pt-2 + et (F4.9.1)

    +

    To fit this model, you need a lagged collection as created previously, except with a longer lag (e.g., lag = 34 days). The next steps are to merge the bands, then reduce with the linear regression reducer.

    +

    Copy and paste the line below to the existing script to create a lagged collection, where the images list stores the two previous images:

    +

    var lagged34 = ee.ImageCollection(lag(landsat8sr, landsat8sr, 34));

    +

    Copy and paste the code below to merge the bands of the lagged collection such that each image has bands at time t and bands at times t - 1,…, t − lag. Note that it’s necessary to filter out any images that don’t have two previous temporal neighbors.

    +

    var merged34 = lagged34.map(merge).map(function(image) { return image.set(‘n’, ee.List(image.get(‘images’))
    +.length());
    }).filter(ee.Filter.gt(‘n’, 1));

    -

    Now, copy and paste the code below to fit the regression model using the linearRegression reducer.

    -

    var arIndependents = ee.List([‘constant’, ‘NDVI_1’, ‘NDVI_2’]);

    -

    var ar2 = merged34
    -   .select(arIndependents.add(dependent))
    -   .reduce(ee.Reducer.linearRegression(arIndependents.length(), 1));

    -

    // Turn the array image into a multi-band image of coefficients.
    -var arCoefficients = ar2.select(‘coefficients’)
    -   .arrayProject([0])
    -   .arrayFlatten([arIndependents]);

    -

    We can compute the fitted values using the expression function in Earth Engine. Because this model is a function of previous pixel values, which may be masked, if any of the inputs to equation F4.9.1 are masked, the output of the equation will also be masked. That’s why you should use an expression here, unlike the previous linear models of time. Copy and paste the code below to compute the fitted values.

    -

    // Compute fitted values.
    -var fittedAR = merged34.map(function(image) {   return image.addBands(
    -       image.expression(           ‘beta0 + beta1 * p1 + beta2 * p2’, {
    -               p1: image.select(‘NDVI_1’),
    -               p2: image.select(‘NDVI_2’),
    -               beta0: arCoefficients.select(‘constant’),
    -               beta1: arCoefficients.select(‘NDVI_1’),
    -               beta2: arCoefficients.select(‘NDVI_2’)
    -           }).rename(‘fitted’));
    -});

    -

    Finally, copy and paste the code below to plot the results (Fig. F4.9.5). We will use a specific point defined as pt. Note the missing values that result from masked data. If you run into computation errors, try commenting the Map.addLayer calls from previous sections to save memory.

    -

    // Create an Earth Engine point object to print the time series chart.
    -var pt = ee.Geometry.Point([-119.0955, 35.9909]);

    -

    print(ui.Chart.image.series(
    -       fittedAR.select([‘fitted’, ‘NDVI’]), pt, ee.Reducer
    -   .mean(), 30)
    -   .setSeriesNames([‘NDVI’, ‘fitted’])
    -   .setOptions({
    -       title: ‘AR(2) model: original and fitted values’,
    -       lineWidth: 1,
    -       pointSize: 3,
    -   }));

    -

    -

    Fig. F4.9.5 Observed NDVI and fitted values at selected point

    +

    Now, copy and paste the code below to fit the regression model using the linearRegression reducer.

    +

    var arIndependents = ee.List([‘constant’, ‘NDVI_1’, ‘NDVI_2’]);

    +

    var ar2 = merged34
    +.select(arIndependents.add(dependent))
    +.reduce(ee.Reducer.linearRegression(arIndependents.length(), 1));

    +
    // Turn the array image into a multi-band image of coefficients.  
+var arCoefficients = ar2.select('coefficients')  
+   .arrayProject([0])  
+   .arrayFlatten([arIndependents]);
    +

    We can compute the fitted values using the expression function in Earth Engine. Because this model is a function of previous pixel values, which may be masked, if any of the inputs to equation F4.9.1 are masked, the output of the equation will also be masked. That’s why you should use an expression here, unlike the previous linear models of time. Copy and paste the code below to compute the fitted values.

    +
    // Compute fitted values.  
+var fittedAR = merged34.map(function(image) {   return image.addBands(  
+       image.expression(           'beta0 + beta1 * p1 + beta2 * p2', {  
+               p1: image.select('NDVI_1'),  
+               p2: image.select('NDVI_2'),  
+               beta0: arCoefficients.select('constant'),  
+               beta1: arCoefficients.select('NDVI_1'),  
+               beta2: arCoefficients.select('NDVI_2')  
+           }).rename('fitted'));  
+});
    +

    Finally, copy and paste the code below to plot the results (Fig. F4.9.5). We will use a specific point defined as pt. Note the missing values that result from masked data. If you run into computation errors, try commenting the Map.addLayer calls from previous sections to save memory.

    +
    // Create an Earth Engine point object to print the time series chart.  
+var pt = ee.Geometry.Point([-119.0955, 35.9909]);  
+  
+print(ui.Chart.image.series(  
+       fittedAR.select(['fitted', 'NDVI']), pt, ee.Reducer  
+   .mean(), 30)  
+   .setSeriesNames(['NDVI', 'fitted'])  
+   .setOptions({  
+       title: 'AR(2) model: original and fitted values',  
+       lineWidth: 1,  
+       pointSize: 3,  
+   }));
    +
    +
    +

    +

    Fig. F4.9.5 Observed NDVI and fitted values at selected point

    +
    +

    At this stage, note that the missing data has become a real problem. Any data point for which at least one of the previous points is masked or missing is also masked.

    @@ -2949,7 +3465,7 @@ Note
    -

    Code Checkpoint F49c. The book’s repository contains a script that shows what your code should look like at this point.

    +

    Code Checkpoint F49c. The book’s repository contains a script that shows what your code should look like at this point.

    It may be possible to avoid this problem by substituting the output from equation F4.9.1 (the modeled value) for the missing or masked data. Unfortunately, the code to make that happen is not straightforward. You can check a solution in the following Code Checkpoint:

    @@ -2963,13 +3479,13 @@ Note
    -

    Code Checkpoint F49d. The book’s repository contains a script that shows what your code should look like at this point.

    +

    Code Checkpoint F49d. The book’s repository contains a script that shows what your code should look like at this point.

    Synthesis

    -

    Assignment 1. Analyze cross-correlation between NDVI and soil moisture, or precipitation and soil moisture, for example. Earth Engine contains different soil moisture datasets in its catalog (e.g., NASA-USDA SMAP, NASA-GLDAS). Try increasing the lagged time and see if it makes any difference. Alternatively, you can pick any other environmental variable/index (e.g., a different vegetation index: EVI instead of NDVI, for example) and analyze its autocorrelation.

    +

    Assignment 1. Analyze cross-correlation between NDVI and soil moisture, or precipitation and soil moisture, for example. Earth Engine contains different soil moisture datasets in its catalog (e.g., NASA-USDA SMAP, NASA-GLDAS). Try increasing the lagged time and see if it makes any difference. Alternatively, you can pick any other environmental variable/index (e.g., a different vegetation index: EVI instead of NDVI, for example) and analyze its autocorrelation.

    Conclusion

    diff --git a/docs/F5.html b/docs/F5.html index a5dd908..c4b0f96 100644 --- a/docs/F5.html +++ b/docs/F5.html @@ -20,6 +20,69 @@ ul.task-list li input[type="checkbox"] { margin: 0 0.8em 0.2em -1.6em; vertical-align: middle; } +pre > code.sourceCode { white-space: pre; position: relative; } +pre > code.sourceCode > span { display: inline-block; line-height: 1.25; } +pre > code.sourceCode > span:empty { height: 1.2em; } +.sourceCode { overflow: visible; } +code.sourceCode > span { color: inherit; text-decoration: inherit; } +div.sourceCode { margin: 1em 0; } +pre.sourceCode { margin: 0; } +@media screen { +div.sourceCode { overflow: auto; } +} +@media print { +pre > code.sourceCode { white-space: pre-wrap; } +pre > code.sourceCode > span { text-indent: -5em; padding-left: 5em; } +} +pre.numberSource code + { counter-reset: source-line 0; } +pre.numberSource code > span + { position: relative; left: -4em; counter-increment: source-line; } +pre.numberSource code > span > a:first-child::before + { content: counter(source-line); + position: relative; left: -1em; text-align: right; vertical-align: baseline; + border: none; display: inline-block; + -webkit-touch-callout: none; -webkit-user-select: none; + -khtml-user-select: none; -moz-user-select: none; + -ms-user-select: none; user-select: none; + padding: 0 4px; width: 4em; + color: #aaaaaa; + } +pre.numberSource { margin-left: 3em; border-left: 1px solid #aaaaaa; padding-left: 4px; } +div.sourceCode + { } +@media screen { +pre > code.sourceCode > span > a:first-child::before { text-decoration: underline; } +} +code span.al { color: #ff0000; font-weight: bold; } /* Alert */ +code span.an { color: #60a0b0; font-weight: bold; font-style: italic; } /* Annotation */ +code span.at { color: #7d9029; } /* Attribute */ +code span.bn { color: #40a070; } /* BaseN */ +code span.bu { color: #008000; } /* BuiltIn */ +code span.cf { color: #007020; font-weight: bold; } /* ControlFlow */ +code span.ch { color: #4070a0; } /* Char */ +code span.cn { color: #880000; } /* Constant */ +code span.co { color: #60a0b0; font-style: italic; } /* Comment */ +code span.cv { color: #60a0b0; font-weight: bold; font-style: italic; } /* CommentVar */ +code span.do { color: #ba2121; font-style: italic; } /* Documentation */ +code span.dt { color: #902000; } /* DataType */ +code span.dv { color: #40a070; } /* DecVal */ +code span.er { color: #ff0000; font-weight: bold; } /* Error */ +code span.ex { } /* Extension */ +code span.fl { color: #40a070; } /* Float */ +code span.fu { color: #06287e; } /* Function */ +code span.im { color: #008000; font-weight: bold; } /* Import */ +code span.in { color: #60a0b0; font-weight: bold; font-style: italic; } /* Information */ +code span.kw { color: #007020; font-weight: bold; } /* Keyword */ +code span.op { color: #666666; } /* Operator */ +code span.ot { color: #007020; } /* Other */ +code span.pp { color: #bc7a00; } /* Preprocessor */ +code span.sc { color: #4070a0; } /* SpecialChar */ +code span.ss { color: #bb6688; } /* SpecialString */ +code span.st { color: #4070a0; } /* String */ +code span.va { color: #19177c; } /* Variable */ +code span.vs { color: #4070a0; } /* VerbatimString */ +code span.wa { color: #60a0b0; font-weight: bold; font-style: italic; } /* Warning */ @@ -229,8 +292,8 @@ gtag('config', 'G-RK9ZLZQ6GL', { 'anonymize_ip': true});
    • 7 Exploring Vectors
        -
      • 7.1 Using Geometry Tools to Create Features in Earth Engine
        • -
      • 7.2 Loading Existing Features and Feature Collections in Earth Engine
        • +
      • 7.1 Using Geometry Tools to Create Features in Earth Engine
        • +
      • 7.2 Loading Existing Features and Feature Collections in Earth Engine
      • 7.3 Importing Features into Earth Engine
      • 8 Raster/Vector Conversions
          -
        • 8.1 Raster to Vector Conversion  +
        • 8.1 Raster to Vector Conversion
        • 9 Zonal Statistics
            -
          • 9.1 Introduction to Theory 
            • +
          • 9.1 Introduction to Theory
          • 9.2 Functions
            • 9.2.1 Function: bufferPoints(radius, bounds)
              • @@ -336,20 +399,20 @@ gtag('config', 'G-RK9ZLZQ6GL', { 'anonymize_ip': true});

              In addition to raster data processing, Earth Engine supports a rich set of vector processing tools. This Part introduces you to the vector framework in Earth Engine, shows you how to create and to import your vector data, and how to combine vector and raster data for analyses.

              7 Exploring Vectors

              -

              ::: {.callout-tip} # Chapter Information

              +

              :::{.callout-tip} # Chapter Information

              Author

              AJ Purdy, Ellen Brock, David Saah

              Overview

              -

              In this chapter, you will learn about features and feature collections and how to use them in conjunction with images and image collections in Earth Engine. Maps are useful for understanding spatial patterns, but scientists often need to extract statistics to answer a question. For example, you may make a false-color composite showing which areas of San Francisco are more “green”—i.e., have more healthy vegetation—than others, but you will likely not be able to directly determine which block in a neighborhood is the most green. This tutorial will demonstrate how to do just that by utilizing vectors.

              -

              As described in Chap. F4.0, an important way to summarize and simplify data in Earth Engine is through the use of reducers. Reducers operating across space were used in Chap. F3.0, for example, to enable image regression between bands. More generally, chapters in Part F3 and Part F4 used reducers mostly to summarize the values across bands or images on a pixel-by-pixel basis. What if you wanted to summarize information within the confines of given spatial elements- for example, within a set of polygons? In this chapter, we will illustrate and explore Earth Engine’s method for doing that, which is through a reduceRegions call.

              +

              In this chapter, you will learn about features and feature collections and how to use them in conjunction with images and image collections in Earth Engine. Maps are useful for understanding spatial patterns, but scientists often need to extract statistics to answer a question. For example, you may make a false-color composite showing which areas of San Francisco are more “green”—i.e., have more healthy vegetation—than others, but you will likely not be able to directly determine which block in a neighborhood is the most green. This tutorial will demonstrate how to do just that by utilizing vectors.

              +

              As described in Chap. F4.0, an important way to summarize and simplify data in Earth Engine is through the use of reducers. Reducers operating across space were used in Chap. F3.0, for example, to enable image regression between bands. More generally, chapters in Part F3 and Part F4 used reducers mostly to summarize the values across bands or images on a pixel-by-pixel basis. What if you wanted to summarize information within the confines of given spatial elements- for example, within a set of polygons? In this chapter, we will illustrate and explore Earth Engine’s method for doing that, which is through a reduceRegions call.

              Learning Outcomes

              • Uploading and working with a shapefile as an asset to use in Earth Engine.
              • Creating a new feature using the geometry tools.
              • Importing and filtering a feature collection in Earth Engine.
              • Using a feature to clip and reduce image values within a geometry.
                • -
              • Use reduceRegions to summarize an image in irregular neighborhoods.
                • +
              • Use reduceRegions to summarize an image in irregular neighborhoods.
              • Exporting calculated data to tables with Tasks.

              Assumes you know how to:

              @@ -358,23 +421,31 @@ gtag('config', 'G-RK9ZLZQ6GL', { 'anonymize_ip': true});
            • Calculate and interpret vegetation indices (Chap. F2.0).
            • Use drawing tools to create points, lines, and polygons (Chap. F2.1).
            -

            Introduction to Theory 

            -

            In the world of geographic information systems (GIS), data is typically thought of in one of two basic data structures: raster and vector. In previous chapters, we have principally been focused on raster data—data using the remote sensing vocabulary of pixels, spatial resolution, images, and image collections. Working within the vector framework is also a crucial skill to master. If you don’t know much about GIS, you can find any number of online explainers of the distinctions between these data types, their strengths and limitations, and analyses using both data types. Being able to move fluidly between a raster conception and a vector conception of the world is powerful, and is facilitated with specialized functions and approaches in Earth Engine.    

            +

            Introduction to Theory

            +

            In the world of geographic information systems (GIS), data is typically thought of in one of two basic data structures: raster and vector. In previous chapters, we have principally been focused on raster data—data using the remote sensing vocabulary of pixels, spatial resolution, images, and image collections. Working within the vector framework is also a crucial skill to master. If you don’t know much about GIS, you can find any number of online explainers of the distinctions between these data types, their strengths and limitations, and analyses using both data types. Being able to move fluidly between a raster conception and a vector conception of the world is powerful, and is facilitated with specialized functions and approaches in Earth Engine.

            For our purposes, you can think of vector data as information represented as points (e.g., locations of sample sites), lines (e.g., railroad tracks), or polygons (e.g., the boundary of a national park or a neighborhood). Line data and polygon data are built up from points: for example, the latitude and longitude of the sample sites, the points along the curve of the railroad tracks, and the corners of the park that form its boundary. These points each have a highly specific location on Earth’s surface, and the vector data formed from them can be used for calculations with respect to other layers. As will be seen in this chapter, for example, a polygon can be used to identify which pixels in an image are contained within its borders. Point-based data have already been used in earlier chapters for filtering image collections by location (see Part F1), and can also be used to extract values from an image at a point or a set of points (see Chap. F5.2). Lines possess the dimension of length and have similar capabilities for filtering image collections and accessing their values along a transect. In addition to using polygons to summarize values within a boundary, they can be used for other, similar purposes—for example, to clip an image.

            -

            As you have seen, raster features in Earth Engine are stored as an Image or as part of an ImageCollection. Using a similar conceptual model, vector data in Earth Engine is stored as a Feature or as part of a FeatureCollection. Features and feature collections provide useful data to filter images and image collections by their location, clip images to a boundary, or statistically summarize the pixel values within a region.

            -

            In the following example, you will use features and feature collections to identify which city block near the University of San Francisco (USF) campus is the most green.

            +

            As you have seen, raster features in Earth Engine are stored as an Image or as part of an ImageCollection. Using a similar conceptual model, vector data in Earth Engine is stored as a Feature or as part of a FeatureCollection. Features and feature collections provide useful data to filter images and image collections by their location, clip images to a boundary, or statistically summarize the pixel values within a region.

            +

            In the following example, you will use features and feature collections to identify which city block near the University of San Francisco (USF) campus is the most green.

    -

    7.1 Using Geometry Tools to Create Features in Earth Engine

    +

    7.1 Using Geometry Tools to Create Features in Earth Engine

    To demonstrate how geometry tools in Earth Engine work, let’s start by creating a point, and two polygons to represent different elements on the USF campus.

    -

    Click on the geometry tools in the top left of the Map pane and create a point feature. Place a new point where USF is located (see Fig. F5.0.1).

    -

    -

    Fig. F5.0.1 Location of the USF campus in San Francisco, California. Your first point should be in this vicinity. The red arrow points to the geometry tools.

    -

    Use Google Maps to search for “Harney Science Center” or “Lo Schiavo Center for Science.” Hover your mouse over the Geometry Imports to find the +new layer menu item and add a new layer to delineate the boundary of a building on campus.

    +

    Click on the geometry tools in the top left of the Map pane and create a point feature. Place a new point where USF is located (see Fig. F5.0.1).

    +
    +
    +

    +

    Fig. F5.0.1 Location of the USF campus in San Francisco, California. Your first point should be in this vicinity. The red arrow points to the geometry tools.

    +
    +
    +

    Use Google Maps to search for “Harney Science Center” or “Lo Schiavo Center for Science.” Hover your mouse over the Geometry Imports to find the +new layer menu item and add a new layer to delineate the boundary of a building on campus.

    Next, create another new layer to represent the entire campus as a polygon.

    -

    After you create these layers, rename the geometry imports at the top of your script. Name the layers usf_point, usf_building, and usf_campus. These names are used within the script shown in Fig. F5.0.2.

    -

    -

    Fig. F5.0.2 Rename the default variable names for each layer in the Imports section of the code at the top of your script

    +

    After you create these layers, rename the geometry imports at the top of your script. Name the layers usf_point, usf_building, and usf_campus. These names are used within the script shown in Fig. F5.0.2.

    +
    +
    +

    +

    Fig. F5.0.2 Rename the default variable names for each layer in the Imports section of the code at the top of your script

    +
    +
    @@ -385,19 +456,20 @@ Note
    -

    Code Checkpoint F50a. The book’s repository contains a script that shows what your code should look like at this point. 

    +

    Code Checkpoint F50a. The book’s repository contains a script that shows what your code should look like at this point.

    -

    7.2 Loading Existing Features and Feature Collections in Earth Engine

    +

    7.2 Loading Existing Features and Feature Collections in Earth Engine

    If you wish to have the exact same geometry imports in this chapter for the rest of this exercise, begin this section using the code at the Code Checkpoint above.

    -

    Next, you will load a city block dataset to determine the amount of vegetation on blocks near USF. The code below imports an existing feature dataset in Earth Engine. The Topologically Integrated Geographic Encoding and Referencing (TIGER) boundaries are census-designated boundaries that are a useful resource when comparing socioeconomic and diversity metrics with environmental datasets in the United States.

    -

    // Import the Census Tiger Boundaries from GEE.
    -var tiger = ee.FeatureCollection(‘TIGER/2010/Blocks’);

    -

    // Add the new feature collection to the map, but do not display.
    -Map.addLayer(tiger, {   ‘color’: ‘black’}, ‘Tiger’, false);

    -

    You should now have the geometry for USF’s campus and a layer added to your map that is not visualized for census blocks across the United States. Next, we will use neighborhood data to spatially filter the TIGER feature collection for blocks near USF’s campus.

    +

    Next, you will load a city block dataset to determine the amount of vegetation on blocks near USF. The code below imports an existing feature dataset in Earth Engine. The Topologically Integrated Geographic Encoding and Referencing (TIGER) boundaries are census-designated boundaries that are a useful resource when comparing socioeconomic and diversity metrics with environmental datasets in the United States.

    +
    // Import the Census Tiger Boundaries from GEE.  
+var tiger = ee.FeatureCollection('TIGER/2010/Blocks');  
+  
+// Add the new feature collection to the map, but do not display.  
+Map.addLayer(tiger, {   'color': 'black'}, 'Tiger', false);
    +

    You should now have the geometry for USF’s campus and a layer added to your map that is not visualized for census blocks across the United States. Next, we will use neighborhood data to spatially filter the TIGER feature collection for blocks near USF’s campus.

    7.3 Importing Features into Earth Engine

    @@ -405,31 +477,44 @@ Map.addLayer(tiger, {   ‘color’: ‘black’}, ‘Tiger’, false)

    7.3.1 Find a Spatial Dataset of San Francisco Neighborhoods

    Use your internet searching skills to locate the “Analysis Neighborhoods” dataset covering San Francisco. This data might be located in a number of places, including DataSF, the City of San Francisco’s public-facing data repository.

    -

    -

    Fig. F5.0.3 DataSF website neighborhood shapefile to download

    -

    After you find the Analysis Neighborhoods layer, click Export and select Shapefile (Fig. F5.0.3). Keep track of where you save the zipped file, as we will load this into Earth Engine. Shapefiles contain vector-based data—points, lines, polygons—and include a number of files, such as the location information, attribute information, and others.

    -

    Extract the folder to your computer. When you open the folder, you will see that there are actually many files. The extensions (.shp, .dbf, .shx, .prj) all provide a different piece of information to display vector-based data. The .shp file provides data on the geometry. The .dbf file provides data about the attributes. The .shx file is an index file. Lastly, the .prj file describes the map projection of the coordinate information for the shapefile. You will need to load all four files to create a new feature asset in Earth Engine.

    +
    +
    +

    +

    Fig. F5.0.3 DataSF website neighborhood shapefile to download

    +
    +
    +

    After you find the Analysis Neighborhoods layer, click Export and select Shapefile (Fig. F5.0.3). Keep track of where you save the zipped file, as we will load this into Earth Engine. Shapefiles contain vector-based data—points, lines, polygons—and include a number of files, such as the location information, attribute information, and others.

    +

    Extract the folder to your computer. When you open the folder, you will see that there are actually many files. The extensions (.shp, .dbf, .shx, .prj) all provide a different piece of information to display vector-based data. The .shp file provides data on the geometry. The .dbf file provides data about the attributes. The .shx file is an index file. Lastly, the .prj file describes the map projection of the coordinate information for the shapefile. You will need to load all four files to create a new feature asset in Earth Engine.

    7.3.2 Upload SF Neighborhoods File as an Asset

    -

    Navigate to the Assets tab (near Scripts). Select New > Table Upload > Shape files (Fig. F5.0.4).

    -

    -

    Fig. F5.0.4 Import an asset as a zipped folder

    +

    Navigate to the Assets tab (near Scripts). Select New > Table Upload > Shape files (Fig. F5.0.4).

    +
    +
    +

    +

    Fig. F5.0.4 Import an asset as a zipped folder

    +
    +

    7.3.3 Select Files and Name Asset

    -

    Click the Select button and then use the file navigator to select the component files of the shapefile structure (i.e., .shp, .dbf, .shx, and .prj) (Fig. F5.0.5). Assign an Asset Name so you can recognize this asset.

    -

    -

    Fig. F5.0.5 Select the four files extracted from the zipped folder. Make sure each file has the same name and that there are no spaces in the file names of the component files of the shapefile structure.

    -

    Uploading the asset may take a few minutes. The status of the upload is presented under the Tasks tab. After your asset has been successfully loaded, click on the asset in the Assets folder and find the collection ID. Copy this text and use it to import the file into your Earth Engine analysis.

    -

    Assign the asset to the table (collection) ID using the script below. Note that you will need to replace ‘path/to/your/asset/assetname’ with the actual path copied in the previous step.

    -

    // Assign the feature collection to the variable sfNeighborhoods.
    -var sfNeighborhoods = ee.FeatureCollection(   ‘path/to/your/asset/assetname’);

    -

    // Print the size of the feature collection.
    -// (Answers the question how many features?)
    -print(sfNeighborhoods.size());
    -Map.addLayer(sfNeighborhoods, {   ‘color’: ‘blue’}, ‘sfNeighborhoods’);

    -

    Note that if you have any trouble with loading the FeatureCollection using the technique above, you can follow directions in the Checkpoint script below to use an existing asset loaded for this exercise.

    +

    Click the Select button and then use the file navigator to select the component files of the shapefile structure (i.e., .shp, .dbf, .shx, and .prj) (Fig. F5.0.5). Assign an Asset Name so you can recognize this asset.

    +
    +
    +

    +

    Fig. F5.0.5 Select the four files extracted from the zipped folder. Make sure each file has the same name and that there are no spaces in the file names of the component files of the shapefile structure.

    +
    +
    +

    Uploading the asset may take a few minutes. The status of the upload is presented under the Tasks tab. After your asset has been successfully loaded, click on the asset in the Assets folder and find the collection ID. Copy this text and use it to import the file into your Earth Engine analysis.

    +

    Assign the asset to the table (collection) ID using the script below. Note that you will need to replace ‘path/to/your/asset/assetname’ with the actual path copied in the previous step.

    +
    // Assign the feature collection to the variable sfNeighborhoods.  
+var sfNeighborhoods = ee.FeatureCollection(   'path/to/your/asset/assetname');  
+  
+// Print the size of the feature collection.  
+// (Answers the question how many features?)  
+print(sfNeighborhoods.size());  
+Map.addLayer(sfNeighborhoods, {   'color': 'blue'}, 'sfNeighborhoods');
    +

    Note that if you have any trouble with loading the FeatureCollection using the technique above, you can follow directions in the Checkpoint script below to use an existing asset loaded for this exercise.

    @@ -440,7 +525,7 @@ Note
    -

    Code Checkpoint F50b. The book’s repository contains a script that shows what your code should look like at this point.

    +

    Code Checkpoint F50b. The book’s repository contains a script that shows what your code should look like at this point.

    @@ -449,39 +534,39 @@ Note

    7.4 Filtering Feature Collections by Attributes

    7.4.1 Filter by Geometry of Another Feature

    -

    First, let’s find the neighborhood associated with USF. Use the first point you created to find the neighborhood that intersects this point; filterBounds is the tool that does that, returning a filtered feature.

    -

    // Filter sfNeighborhoods by USF.
    -var usfNeighborhood = sfNeighborhoods.filterBounds(usf_point);

    +

    First, let’s find the neighborhood associated with USF. Use the first point you created to find the neighborhood that intersects this point; filterBounds is the tool that does that, returning a filtered feature.

    +
    // Filter sfNeighborhoods by USF.  
+var usfNeighborhood = sfNeighborhoods.filterBounds(usf_point);

    Now, filter the blocks layer by USF’s neighborhood and visualize it on the map.

    -

    // Filter the Census blocks by the boundary of the neighborhood layer.
    -var usfTiger = tiger.filterBounds(usfNeighborhood);
    -Map.addLayer(usfTiger, {}, ‘usf_Tiger’);

    +
    // Filter the Census blocks by the boundary of the neighborhood layer.  
+var usfTiger = tiger.filterBounds(usfNeighborhood);  
+Map.addLayer(usfTiger, {}, 'usf_Tiger');

    7.4.2 Filter by Feature (Attribute) Properties

    -

    In addition to filtering a FeatureCollection by the location of another feature, you can also filter it by its properties. First, let’s print the usfTiger variable to the Console and inspect the object.

    +

    In addition to filtering a FeatureCollection by the location of another feature, you can also filter it by its properties. First, let’s print the usfTiger variable to the Console and inspect the object.

    print(usfTiger);

    -

    You can click on the feature collection name in the Console to uncover more information about the dataset. Click on the columns to learn about what attribute information is contained in this dataset. You will notice this feature collection contains information on both housing (‘housing10’) and population (‘pop10’).

    +

    You can click on the feature collection name in the Console to uncover more information about the dataset. Click on the columns to learn about what attribute information is contained in this dataset. You will notice this feature collection contains information on both housing (‘housing10’) and population (‘pop10’).

    Now you will filter for blocks with just the right amount of housing units. You don’t want it too dense, nor do you want too few neighbors.

    Filter the blocks to have fewer than 250 housing units.

    -

    // Filter for census blocks by housing units.
    -var housing10_l250 = usfTiger
    -   .filter(ee.Filter.lt(‘housing10’, 250));

    +
    // Filter for census blocks by housing units.  
+var housing10_l250 = usfTiger  
+   .filter(ee.Filter.lt('housing10', 250));

    Now filter the already-filtered blocks to have more than 50 housing units.

    -

    var housing10_g50_l250 = housing10_l250.filter(ee.Filter.gt(   ‘housing10’, 50));

    +

    var housing10_g50_l250 = housing10_l250.filter(ee.Filter.gt( ‘housing10’, 50));

    Now, let’s visualize what this looks like.

    -

    Map.addLayer(housing10_g50_l250, {   ‘color’: ‘Magenta’}, ‘housing’);

    -

    We have combined spatial and attribute information to narrow the set to only those blocks that meet our criteria of having between 50 and 250 housing units.

    +

    Map.addLayer(housing10_g50_l250, { ‘color’: ‘Magenta’}, ‘housing’);

    +

    We have combined spatial and attribute information to narrow the set to only those blocks that meet our criteria of having between 50 and 250 housing units.

    @@ -503,61 +588,66 @@ Note

    7.5 Reducing Images Using Feature Geometry

    Now that we have identified the blocks around USF’s campus that have the right housing density, let’s find which blocks are the greenest.

    -

    The Normalized Difference Vegetation Index (NDVI), presented in detail in Chap. F2.0, is often used to compare the greenness of pixels in different locations. Values on land range from 0 to 1, with values closer to 1 representing healthier and greener vegetation than values near 0.

    +

    The Normalized Difference Vegetation Index (NDVI), presented in detail in Chap. F2.0, is often used to compare the greenness of pixels in different locations. Values on land range from 0 to 1, with values closer to 1 representing healthier and greener vegetation than values near 0.

    7.5.1 Create an NDVI Image

    -

    The code below imports the Landsat 8 ImageCollection as landsat8. Then, the code filters for images in 2021. Lastly, the code sorts the images from 2021 to find the least cloudy day.

    -

    // Import the Landsat 8 TOA image collection.
    -var landsat8 = ee.ImageCollection(‘LANDSAT/LC08/C02/T1_TOA’);

    -

    // Get the least cloudy image in 2015.
    -var image = ee.Image(
    -   landsat8
    -   .filterBounds(usf_point)
    -   .filterDate(‘2015-01-01’, ‘2015-12-31’)
    -   .sort(‘CLOUD_COVER’)
    -   .first());

    -

    The next section of code assigns the near-infrared band (B5) to variable nir and assigns the red band (B4) to red. Then the bands are combined together to compute NDVI as (nir − red)/(nir + red).

    -

    var nir = image.select(‘B5’);
    -var red = image.select(‘B4’);
    -var ndvi = nir.subtract(red).divide(nir.add(red)).rename(‘NDVI’);

    +

    The code below imports the Landsat 8 ImageCollection as landsat8. Then, the code filters for images in 2021. Lastly, the code sorts the images from 2021 to find the least cloudy day.

    +
    // Import the Landsat 8 TOA image collection.  
+var landsat8 = ee.ImageCollection('LANDSAT/LC08/C02/T1_TOA');  
+  
+// Get the least cloudy image in 2015.  
+var image = ee.Image(  
+   landsat8  
+   .filterBounds(usf_point)  
+   .filterDate('2015-01-01', '2015-12-31')  
+   .sort('CLOUD_COVER')  
+   .first());
    +

    The next section of code assigns the near-infrared band (B5) to variable nir and assigns the red band (B4) to red. Then the bands are combined together to compute NDVI as (nir − red)/(nir + red).

    +

    var nir = image.select(‘B5’);
    +var red = image.select(‘B4’);
    +var ndvi = nir.subtract(red).divide(nir.add(red)).rename(‘NDVI’);

    7.5.2 Clip the NDVI Image to the Blocks Near USF

    Next, you will clip the NDVI layer to only show NDVI over USF’s neighborhood.

    The first section of code provides visualization settings.

    -

    var ndviParams = {
    -   min: -1,
    -   max: 1,
    -   palette: [‘blue’, ‘white’, ‘green’]
    +

    var ndviParams = {
    +min: -1,
    +max: 1,
    +palette: [‘blue’, ‘white’, ‘green’]
    };

    The second block of code clips the image to our filtered housing layer.

    -

    var ndviUSFblocks = ndvi.clip(housing10_g50_l250);
    +

    var ndviUSFblocks = ndvi.clip(housing10_g50_l250);
    Map.addLayer(ndviUSFblocks, ndviParams, ‘NDVI image’);
    Map.centerObject(usf_point, 14);

    -

    The NDVI map for all of San Francisco is interesting, and shows variability across the region. Now, let’s compute mean NDVI values for each block of the city.

    +

    The NDVI map for all of San Francisco is interesting, and shows variability across the region. Now, let’s compute mean NDVI values for each block of the city.

    7.5.3 Compute NDVI Statistics by Block

    -

    The code below uses the clipped image ndviUSFblocks and computes the mean NDVI value within each boundary. The scale provides a spatial resolution for the mean values to be computed on.

    -

    // Reduce image by feature to compute a statistic e.g. mean, max, min etc.
    -var ndviPerBlock = ndviUSFblocks.reduceRegions({
    -   collection: housing10_g50_l250,
    -   reducer: ee.Reducer.mean(),
    -   scale: 30,
    -});

    -

    Now we’ll use Earth Engine to find out which block is greenest.  

    +

    The code below uses the clipped image ndviUSFblocks and computes the mean NDVI value within each boundary. The scale provides a spatial resolution for the mean values to be computed on.

    +
    // Reduce image by feature to compute a statistic e.g. mean, max, min etc.  
+var ndviPerBlock = ndviUSFblocks.reduceRegions({  
+   collection: housing10_g50_l250,  
+   reducer: ee.Reducer.mean(),  
+   scale: 30,  
+});
    +

    Now we’ll use Earth Engine to find out which block is greenest.

    7.5.4 Export Table of NDVI Data by Block from Earth Engine to Google Drive

    Just as we loaded a feature into Earth Engine, we can export information from Earth Engine. Here, we will export the NDVI data, summarized by block, from Earth Engine to a Google Drive space so that we can interpret it in a program like Google Sheets or Excel.

    -

    // Get a table of data out of Google Earth Engine.
    -Export.table.toDrive({
    -   collection: ndviPerBlock,
    -   description: ‘NDVI_by_block_near_USF’
    -});

    -

    When you run this code, you will notice that you have the Tasks tab highlighted on the top right of the Earth Engine Code Editor (Fig. F5.0.6). You will be prompted to name the directory when exporting the data.

    -

    -

    Fig. F5.0.6 Under the Tasks tab, select Run to initiate download

    +
    // Get a table of data out of Google Earth Engine.  
+Export.table.toDrive({  
+   collection: ndviPerBlock,  
+   description: 'NDVI_by_block_near_USF'  
+});
    +

    When you run this code, you will notice that you have the Tasks tab highlighted on the top right of the Earth Engine Code Editor (Fig. F5.0.6). You will be prompted to name the directory when exporting the data.

    +
    +
    +

    +

    Fig. F5.0.6 Under the Tasks tab, select Run to initiate download

    +
    +

    After you run the task, the file will be saved to your Google Drive. You have now brought a feature into Earth Engine and also exported data from Earth Engine.

    @@ -569,7 +659,7 @@ Note
    -

    Code Checkpoint F50d. The book’s repository contains a script that shows what your code should look like at this point. 

    +

    Code Checkpoint F50d. The book’s repository contains a script that shows what your code should look like at this point.

    @@ -579,15 +669,15 @@ Note

    You are already familiar with filtering datasets by their attributes. Now you will sort a table and select the first element of the table.

    ndviPerBlock = ndviPerBlock.select([‘blockid10’, ‘mean’]);
    print(‘ndviPerBlock’, ndviPerBlock);
    -var ndviPerBlockSorted = ndviPerBlock.sort(‘mean’, false);
    -var ndviPerBlockSortedFirst = ee.Feature(ndviPerBlock.sort(‘mean’,       false) //Sort by NDVI mean in descending order.   .first()); //Select the block with the highest NDVI.
    +var ndviPerBlockSorted = ndviPerBlock.sort(‘mean’, false);
    +var ndviPerBlockSortedFirst = ee.Feature(ndviPerBlock.sort(‘mean’, false) //Sort by NDVI mean in descending order. .first()); //Select the block with the highest NDVI.
    print(‘ndviPerBlockSortedFirst’, ndviPerBlockSortedFirst);

    -

    If you expand the feature of ndviPerBlockSortedFirst in the Console, you will be able to identify the blockid10 value of the greenest block and the mean NDVI value for that area.

    -

    Another way to look at the data is by exporting the data to a table. Open the table using Google Sheets or Excel. You should see a column titled “mean.” Sort the mean column in descending order from highest NDVI to lowest NDVI, then use the blockid10 attribute to filter our feature collection one last time and display the greenest block near USF.

    -

    // Now filter by block and show on map!
    -var GreenHousing = usfTiger.filter(ee.Filter.eq(‘blockid10’,
    -‘###’)); //< Put your id here prepend a 0!
    -Map.addLayer(GreenHousing, {   ‘color’: ‘yellow’}, ‘Green Housing!’);

    +

    If you expand the feature of ndviPerBlockSortedFirst in the Console, you will be able to identify the blockid10 value of the greenest block and the mean NDVI value for that area.

    +

    Another way to look at the data is by exporting the data to a table. Open the table using Google Sheets or Excel. You should see a column titled “mean.” Sort the mean column in descending order from highest NDVI to lowest NDVI, then use the blockid10 attribute to filter our feature collection one last time and display the greenest block near USF.

    +
    // Now filter by block and show on map!  
+var GreenHousing = usfTiger.filter(ee.Filter.eq('blockid10',  
+'###')); //< Put your id here prepend a 0!  
+Map.addLayer(GreenHousing, {   'color': 'yellow'}, 'Green Housing!');
    @@ -598,17 +688,17 @@ Note
    -

    Code Checkpoint F50e. The book’s repository contains a script that shows what your code should look like at this point. 

    +

    Code Checkpoint F50e. The book’s repository contains a script that shows what your code should look like at this point.

    Synthesis

    Now it’s your turn to use both feature classes and to reduce data using a geographic boundary. Create a new script for an area of interest and accomplish the following assignments.

    -

    Assignment 1. Create a study area map zoomed to a certain feature class that you made.

    -

    Assignment 2. Filter one feature collection using feature properties.

    -

    Assignment 3. Filter one feature collection based on another feature’s location in space.

    -

    Assignment 4. Reduce one image to the geometry of a feature in some capacity; e.g., extract a mean value or a value at a point.

    +

    Assignment 1. Create a study area map zoomed to a certain feature class that you made.

    +

    Assignment 2. Filter one feature collection using feature properties.

    +

    Assignment 3. Filter one feature collection based on another feature’s location in space.

    +

    Assignment 4. Reduce one image to the geometry of a feature in some capacity; e.g., extract a mean value or a value at a point.

    Conclusion

    @@ -633,7 +723,6 @@ Chapter Information

    Overview

    -

     

    The purpose of this chapter is to review methods of converting between raster and vector data formats, and to understand the circumstances in which this is useful. By way of example, this chapter focuses on topographic elevation and forest cover change in Colombia, but note that these are generic methods that can be applied in a wide variety of situations.

    @@ -642,157 +731,180 @@ Chapter Information
  • Understanding raster and vector data in Earth Engine and their differing properties.
  • Knowing how and why to convert from raster to vector.
  • Knowing how and why to convert from vector to raster.
    • -
  • Write a function and map it over a FeatureCollection.
    • +
  • Write a function and map it over a FeatureCollection.

    • Assumes you know how to:

    • Import images and image collections, filter, and visualize (Part F1).
      • -
    • Understand distinctions among Image, ImageCollection, Feature and FeatureCollection Earth Engine objects (Part F1, Part F2, Part F5).
      • +
    • Understand distinctions among Image, ImageCollection, Feature and FeatureCollection Earth Engine objects (Part F1, Part F2, Part F5).
    • Perform basic image analysis: select bands, compute indices, create masks (Part F2).
    • Perform image morphological operations (Chap. F3.2).
      • -
    • Understand the filter, map, reduce paradigm (Chap. F4.0).
      • -
    • Write a function and map it over an ImageCollection (Chap. F4.0).
      • -
    • Use reduceRegions to summarize an image in irregular shapes (Chap. F5.0).
      • +
    • Understand the filter, map, reduce paradigm (Chap. F4.0).
      • +
    • Write a function and map it over an ImageCollection (Chap. F4.0).
      • +
    • Use reduceRegions to summarize an image in irregular shapes (Chap. F5.0).

    Introduction

    -

    Raster data consists of regularly spaced pixels arranged into rows and columns, familiar as the format of satellite images. Vector data contains geometry features (i.e., points, lines, and polygons) describing locations and areas. Each data format has its advantages, and both will be encountered as part of GIS operations.

    +

    Raster data consists of regularly spaced pixels arranged into rows and columns, familiar as the format of satellite images. Vector data contains geometry features (i.e., points, lines, and polygons) describing locations and areas. Each data format has its advantages, and both will be encountered as part of GIS operations.

    Raster and vector data are commonly combined (e.g., extracting image information for a given location or clipping an image to an area of interest); however, there are also situations in which conversion between the two formats is useful. In making such conversions, it is important to consider the key advantages of each format. Rasters can store data efficiently where each pixel has a numerical value, while vector data can more effectively represent geometric features where homogenous areas have shared properties. Each format lends itself to distinctive analytical operations, and combining them can be powerful.

    In this exercise, we’ll use topographic elevation and forest change images in Colombia as well as a protected area feature collection to practice the conversion between raster and vector formats, and to identify situations in which this is worthwhile.

    -

    8.1 Raster to Vector Conversion 

    +

    8.1 Raster to Vector Conversion

    8.1.1 Raster to Polygons

    -

    In this section we will convert an elevation image (raster) to a feature collection (vector). We will start by loading the Global Multi-Resolution Terrain Elevation Data 2010 and the Global Administrative Unit Layers 2015 dataset to focus on Colombia. The elevation image is a raster at 7.5 arc-second spatial resolution containing a continuous measure of elevation in meters in each pixel.

    -

    // Load raster (elevation) and vector (colombia) datasets.
    -var elevation = ee.Image(‘USGS/GMTED2010’).rename(‘elevation’);
    -var colombia = ee.FeatureCollection(       ‘FAO/GAUL_SIMPLIFIED_500m/2015/level0’)
    -   .filter(ee.Filter.equals(‘ADM0_NAME’, ‘Colombia’));

    -

    // Display elevation image.
    -Map.centerObject(colombia, 7);
    -Map.addLayer(elevation, {
    -   min: 0,
    -   max: 4000}, ‘Elevation’);

    +

    In this section we will convert an elevation image (raster) to a feature collection (vector). We will start by loading the Global Multi-Resolution Terrain Elevation Data 2010 and the Global Administrative Unit Layers 2015 dataset to focus on Colombia. The elevation image is a raster at 7.5 arc-second spatial resolution containing a continuous measure of elevation in meters in each pixel.

    +
    // Load raster (elevation) and vector (colombia) datasets.  
+var elevation = ee.Image('USGS/GMTED2010').rename('elevation');  
+var colombia = ee.FeatureCollection(       'FAO/GAUL_SIMPLIFIED_500m/2015/level0')  
+   .filter(ee.Filter.equals('ADM0_NAME', 'Colombia'));  
+  
+// Display elevation image.  
+Map.centerObject(colombia, 7);  
+Map.addLayer(elevation, {  
+   min: 0,  
+   max: 4000}, 'Elevation');

    When converting an image to a feature collection, we will aggregate the categorical elevation values into a set of categories to create polygon shapes of connected pixels with similar elevations. For this exercise, we will create four zones of elevation by grouping the altitudes to 0-100 m = 0, 100–200 m = 1, 200–500 m = 2, and >500 m = 3.

    -

    // Initialize image with zeros and define elevation zones.
    -var zones = ee.Image(0)
    -   .where(elevation.gt(100), 1)
    -   .where(elevation.gt(200), 2)
    -   .where(elevation.gt(500), 3);

    -

    // Mask pixels below sea level (<= 0 m) to retain only land areas.
    -// Name the band with values 0-3 as ‘zone’.
    -zones = zones.updateMask(elevation.gt(0)).rename(‘zone’);

    -

    Map.addLayer(zones, {
    -   min: 0,
    -   max: 3,
    -   palette: [‘white’, ‘yellow’, ‘lime’, ‘green’],
    -   opacity: 0.7}, ‘Elevation zones’);

    -

    We will convert this zonal elevation image in Colombia to polygon shapes, which is a vector format (termed a FeatureCollection in Earth Engine), using the ee.Image.reduceToVectors method. This will create polygons delineating connected pixels with the same value. In doing so, we will use the same projection and spatial resolution as the image. Please note that loading the vectorized image in the native resolution (231.92 m) takes time to execute. For faster visualization, we set a coarse scale of 1,000 m.

    -

    var projection = elevation.projection();
    -var scale = elevation.projection().nominalScale();

    -

    var elevationVector = zones.reduceToVectors({
    -   geometry: colombia.geometry(),
    -   crs: projection,
    -   scale: 1000, // scale   geometryType: ‘polygon’,
    -   eightConnected: false,
    -   labelProperty: ‘zone’,
    -   bestEffort: true,
    -   maxPixels: 1e13,
    -   tileScale: 3 // In case of error.
    +

    // Initialize image with zeros and define elevation zones.  
+var zones = ee.Image(0)  
+   .where(elevation.gt(100), 1)  
+   .where(elevation.gt(200), 2)  
+   .where(elevation.gt(500), 3);  
+  
+// Mask pixels below sea level (<= 0 m) to retain only land areas.  
+// Name the band with values 0-3 as 'zone'.  
+zones = zones.updateMask(elevation.gt(0)).rename('zone');  
+  
+Map.addLayer(zones, {  
+   min: 0,  
+   max: 3,  
+   palette: ['white', 'yellow', 'lime', 'green'],  
+   opacity: 0.7}, 'Elevation zones');
    +

    We will convert this zonal elevation image in Colombia to polygon shapes, which is a vector format (termed a FeatureCollection in Earth Engine), using the ee.Image.reduceToVectors method. This will create polygons delineating connected pixels with the same value. In doing so, we will use the same projection and spatial resolution as the image. Please note that loading the vectorized image in the native resolution (231.92 m) takes time to execute. For faster visualization, we set a coarse scale of 1,000 m.

    +

    var projection = elevation.projection();
    +var scale = elevation.projection().nominalScale();

    +

    var elevationVector = zones.reduceToVectors({
    +geometry: colombia.geometry(),
    +crs: projection,
    +scale: 1000, // scale geometryType: ‘polygon’,
    +eightConnected: false,
    +labelProperty: ‘zone’,
    +bestEffort: true,
    +maxPixels: 1e13,
    +tileScale: 3 // In case of error.
    });

    print(elevationVector.limit(10));

    -

    var elevationDrawn = elevationVector.draw({
    -   color: ‘black’,
    -   strokeWidth: 1
    +

    var elevationDrawn = elevationVector.draw({
    +color: ‘black’,
    +strokeWidth: 1
    });
    Map.addLayer(elevationDrawn, {}, ‘Elevation zone polygon’);

    -

    -

    Fig. F5.1.1 Raster-based elevation (top left) and zones (top right), vectorized elevation zones overlaid on the raster (bottom-left) and vectorized elevation zones only (bottom-right)

    -

    You may have realized that polygons consist of complex lines, including some small polygons with just one pixel. That happens when there are no surrounding pixels of the same elevation zone. You may not need a vector map with such details—if, for instance, you want to produce a regional or global map. We can use a morphological reducer focalMode to simplify the shape by defining a neighborhood size around a pixel. In this example, we will set the kernel radius as four pixels. This operation makes the resulting polygons look much smoother, but less precise (Fig. F5.1.2).

    -

    var zonesSmooth = zones.focalMode(4, ‘square’);

    +
    +
    +

    +

    Fig. F5.1.1 Raster-based elevation (top left) and zones (top right), vectorized elevation zones overlaid on the raster (bottom-left) and vectorized elevation zones only (bottom-right)

    +
    +
    +

    You may have realized that polygons consist of complex lines, including some small polygons with just one pixel. That happens when there are no surrounding pixels of the same elevation zone. You may not need a vector map with such details—if, for instance, you want to produce a regional or global map. We can use a morphological reducer focalMode to simplify the shape by defining a neighborhood size around a pixel. In this example, we will set the kernel radius as four pixels. This operation makes the resulting polygons look much smoother, but less precise (Fig. F5.1.2).

    +

    var zonesSmooth = zones.focalMode(4, ‘square’);

    zonesSmooth = zonesSmooth.reproject(projection.atScale(scale));

    Map.addLayer(zonesSmooth, {
    -   min: 1,
    -   max: 3,
    -   palette: [‘yellow’, ‘lime’, ‘green’],
    -   opacity: 0.7}, ‘Elevation zones (smooth)’);

    -

    var elevationVectorSmooth = zonesSmooth.reduceToVectors({
    -   geometry: colombia.geometry(),
    -   crs: projection,
    -   scale: scale,
    -   geometryType: ‘polygon’,
    -   eightConnected: false,
    -   labelProperty: ‘zone’,
    -   bestEffort: true,
    -   maxPixels: 1e13,
    -   tileScale: 3
    +min: 1,
    +max: 3,
    +palette: [‘yellow’, ‘lime’, ‘green’],
    +opacity: 0.7}, ‘Elevation zones (smooth)’);

    +

    var elevationVectorSmooth = zonesSmooth.reduceToVectors({
    +geometry: colombia.geometry(),
    +crs: projection,
    +scale: scale,
    +geometryType: ‘polygon’,
    +eightConnected: false,
    +labelProperty: ‘zone’,
    +bestEffort: true,
    +maxPixels: 1e13,
    +tileScale: 3
    });

    -

    var smoothDrawn = elevationVectorSmooth.draw({
    -   color: ‘black’,
    -   strokeWidth: 1
    +

    var smoothDrawn = elevationVectorSmooth.draw({
    +color: ‘black’,
    +strokeWidth: 1
    });
    Map.addLayer(smoothDrawn, {}, ‘Elevation zone polygon (smooth)’);

    -

    We can see now that the polygons have more distinct shapes with many fewer small polygons in the new map (Fig. F5.1.2). It is important to note that when you use methods like focalMode (or other, similar methods such as connectedComponents and connectedPixelCount), you need to reproject according to the original image in order to display properly with zoom using the interactive Code Editor.

    +

    We can see now that the polygons have more distinct shapes with many fewer small polygons in the new map (Fig. F5.1.2). It is important to note that when you use methods like focalMode (or other, similar methods such as connectedComponents and connectedPixelCount), you need to reproject according to the original image in order to display properly with zoom using the interactive Code Editor.

    -

    -

    Fig. F5.1.2 Before (left) and after (right) applying focalMode

    +
    +
    +

    +

    Fig. F5.1.2 Before (left) and after (right) applying focalMode

    +
    +

    8.1.2 Raster to Points

    Lastly, we will convert a small part of this elevation image into a point vector dataset. For this exercise, we will use the same example and build on the code from the previous subsection. This might be useful when you want to use geospatial data in a tabular format in combination with other conventional datasets such as economic indicators (Fig. F5.1.3).

    -

    -

    Fig. F5.1.3 Elevation point values with latitude and longitude

    -

    The easiest way to do this is to use sample while activating the geometries parameter. This will extract the points at the centroid of the elevation pixel.

    -

    var geometry = ee.Geometry.Polygon([
    -   [-89.553, -0.929],
    -   [-89.436, -0.929],
    -   [-89.436, -0.866],
    -   [-89.553, -0.866],
    -   [-89.553, -0.929]
    +

    +
    +

    +

    Fig. F5.1.3 Elevation point values with latitude and longitude

    +
    +
    +

    The easiest way to do this is to use sample while activating the geometries parameter. This will extract the points at the centroid of the elevation pixel.

    +

    var geometry = ee.Geometry.Polygon([
    +[-89.553, -0.929],
    +[-89.436, -0.929],
    +[-89.436, -0.866],
    +[-89.553, -0.866],
    +[-89.553, -0.929]
    ]);

    -

    // To zoom into the area, un-comment and run below
    -// Map.centerObject(geometry,12);
    -Map.addLayer(geometry, {}, ‘Areas to extract points’);

    -

    var elevationSamples = elevation.sample({
    -   region: geometry,
    -   projection: projection,
    -   scale: scale,
    -   geometries: true,
    -});

    -

    Map.addLayer(elevationSamples, {}, ‘Points extracted’);

    -

    // Add three properties to the output table:
    -// ‘Elevation’, ‘Longitude’, and ‘Latitude’.
    -elevationSamples = elevationSamples.map(function(feature) {   var geom = feature.geometry().coordinates();   return ee.Feature(null, {       ‘Elevation’: ee.Number(feature.get(           ‘elevation’)),       ‘Long’: ee.Number(geom.get(0)),       ‘Lat’: ee.Number(geom.get(1))
    -   });
    -});

    -

    // Export as CSV.
    -Export.table.toDrive({
    -   collection: elevationSamples,
    -   description: ‘extracted_points’,
    -   fileFormat: ‘CSV’
    -});

    -

    We can also extract sample points per elevation zone. Below is an example of extracting 10 randomly selected points per elevation zone (Fig. F5.1.4). You can also set different values for each zone using classValues and classPoints parameters to modify the sampling intensity in each class. This may be useful, for instance, to generate point samples for a validation effort.

    -

    var elevationSamplesStratified = zones.stratifiedSample({
    -   numPoints: 10,
    -   classBand: ‘zone’,
    -   region: geometry,
    -   scale: scale,
    -   projection: projection,
    -   geometries: true
    +

    // To zoom into the area, un-comment and run below  
+// Map.centerObject(geometry,12);  
+Map.addLayer(geometry, {}, 'Areas to extract points');  
+  
+var elevationSamples = elevation.sample({  
+   region: geometry,  
+   projection: projection,  
+   scale: scale,  
+   geometries: true,  
+});  
+  
+Map.addLayer(elevationSamples, {}, 'Points extracted');  
+  
+// Add three properties to the output table:  
+// 'Elevation', 'Longitude', and 'Latitude'.  
+elevationSamples = elevationSamples.map(function(feature) {   var geom = feature.geometry().coordinates();   return ee.Feature(null, {       'Elevation': ee.Number(feature.get(           'elevation')),       'Long': ee.Number(geom.get(0)),       'Lat': ee.Number(geom.get(1))  
+   });  
+});  
+  
+// Export as CSV.  
+Export.table.toDrive({  
+   collection: elevationSamples,  
+   description: 'extracted_points',  
+   fileFormat: 'CSV'  
+});
    +

    We can also extract sample points per elevation zone. Below is an example of extracting 10 randomly selected points per elevation zone (Fig. F5.1.4). You can also set different values for each zone using classValues and classPoints parameters to modify the sampling intensity in each class. This may be useful, for instance, to generate point samples for a validation effort.

    +

    var elevationSamplesStratified = zones.stratifiedSample({
    +numPoints: 10,
    +classBand: ‘zone’,
    +region: geometry,
    +scale: scale,
    +projection: projection,
    +geometries: true
    });

    Map.addLayer(elevationSamplesStratified, {}, ‘Stratified samples’);

    -

    -

    Fig. F5.1.4 Stratified sampling over different elevation zones

    +
    +
    +

    +

    Fig. F5.1.4 Stratified sampling over different elevation zones

    +
    +
    @@ -803,119 +915,146 @@ Note
    -

    Code Checkpoint F51a. The book’s repository contains a script that shows what your code should look like at this point.

    +

    Code Checkpoint F51a. The book’s repository contains a script that shows what your code should look like at this point.

    ##3. A More Complex Example

    In this section we’ll use two global datasets, one to represent raster formats and the other vectors:

      -
    • The Global Forest Change (GFC) dataset: a raster dataset describing global tree cover and change for 2001–present.
      • +
    • The Global Forest Change (GFC) dataset: a raster dataset describing global tree cover and change for 2001–present.
    • The World Protected Areas Database: a vector database of global protected areas.

    The objective will be to combine these two datasets to quantify rates of deforestation in protected areas in the “arc of deforestation” of the Colombian Amazon. The datasets can be loaded into Earth Engine with the following code:

    -

    // Read input data.
    -// Note: these datasets are periodically updated.
    -// Consider searching the Data Catalog for newer versions.
    -var gfc = ee.Image(‘UMD/hansen/global_forest_change_2020_v1_8’);
    -var wdpa = ee.FeatureCollection(‘WCMC/WDPA/current/polygons’);

    -

    // Print assets to show available layers and properties.
    -print(gfc);
    -print(wdpa.limit(10)); // Show first 10 records.

    -

    The GFC dataset (first presented in detail in Chap. F1.1) is a global set of rasters that quantify tree cover and change for the period beginning in 2001. We’ll use a single image from this dataset:

    -
      -
    • ‘lossyear’: a categorical raster of forest loss (1–20, corresponding to deforestation for the period 2001–2020), and 0 for no change
      • -
    -

    The World Database on Protected Areas (WDPA) is a harmonized dataset of global terrestrial and marine protected area locations, along with details on the classification and management of each. In addition to protected area outlines, we’ll use two fields from this database:

    -
      -
    • ‘NAME’’: the name of each protected area
      • -
    • ‘WDPA_PID’: a unique numerical ID for each protected area
      • -
    -

    To begin with, we’ll focus on forest change dynamics in ‘La Paya’, a small protected area in the Colombian Amazon. We’ll first visualize these data using the paint command, which is discussed in more detail in Chap. F5.3:

    -

    // Display deforestation.
    -var deforestation = gfc.select(‘lossyear’);

    -

    Map.addLayer(deforestation, {
    -   min: 1,
    -   max: 20,
    -   palette: [‘yellow’, ‘orange’, ‘red’]
    -}, ‘Deforestation raster’);

    -

    // Display WDPA data.
    -var protectedArea = wdpa.filter(ee.Filter.equals(‘NAME’, ‘La Paya’));

    -

    // Display protected area as an outline (see F5.3 for paint()).
    -var protectedAreaOutline = ee.Image().byte().paint({
    -   featureCollection: protectedArea,
    -   color: 1,
    -   width: 3
    -});

    -

    Map.addLayer(protectedAreaOutline, {
    -   palette: ‘white’}, ‘Protected area’);

    -

    // Set up map display.
    -Map.centerObject(protectedArea);
    -Map.setOptions(‘SATELLITE’);

    +
    // Read input data.  
+// Note: these datasets are periodically updated.  
+// Consider searching the Data Catalog for newer versions.  
+var gfc = ee.Image('UMD/hansen/global_forest_change_2020_v1_8');  
+var wdpa = ee.FeatureCollection('WCMC/WDPA/current/polygons');  
+  
+// Print assets to show available layers and properties.  
+print(gfc);  
+print(wdpa.limit(10)); // Show first 10 records.
+
+The GFC dataset (first presented in detail in Chap. F1.1) is a global set of rasters that quantify tree cover and change for the period beginning in 2001. We’ll use a single image from this dataset:
+
+*   'lossyear': a categorical raster of forest loss (1–20, corresponding to deforestation for the period 2001–2020), and 0 for no change
+
+The World Database on Protected Areas (WDPA) is a harmonized dataset of global terrestrial and marine protected area locations, along with details on the classification and management of each. In addition to protected area outlines, we’ll use two fields from this database:
+
+*   'NAME'’: the name of each protected area
+*   ‘WDPA_PID’: a unique numerical ID for each protected area
+
+To begin with, we’ll focus on forest change dynamics in ‘La Paya’, a small protected area in the Colombian Amazon. We’ll first visualize these data using the paint command, which is discussed in more detail in Chap. F5.3:
+
+// Display deforestation.  
+var deforestation = gfc.select('lossyear');  
+  
+Map.addLayer(deforestation, {  
+   min: 1,  
+   max: 20,  
+   palette: ['yellow', 'orange', 'red']  
+}, 'Deforestation raster');  
+  
+// Display WDPA data.  
+var protectedArea = wdpa.filter(ee.Filter.equals('NAME', 'La Paya'));  
+  
+// Display protected area as an outline (see F5.3 for paint()).  
+var protectedAreaOutline = ee.Image().byte().paint({  
+   featureCollection: protectedArea,  
+   color: 1,  
+   width: 3  
+});  
+  
+Map.addLayer(protectedAreaOutline, {  
+   palette: 'white'}, 'Protected area');  
+  
+// Set up map display.  
+Map.centerObject(protectedArea);  
+Map.setOptions('SATELLITE');

    This will display the boundary of the La Paya protected area and deforestation in the region (Fig. F5.1.5).

    -

    -

    Fig. F5.1.5 View of the La Paya protected area in the Colombian Amazon (in white), and deforestation over the period 2001–2020 (in yellows and reds, with darker colors indicating more recent changes)

    -

    We can use Earth Engine to convert the deforestation raster to a set of polygons. The deforestation data are appropriate for this transformation as each deforestation event is labeled categorically by year, and change events are spatially contiguous. This is performed in Earth Engine using the ee.Image.reduceToVectors method, as described earlier in this section.  

    -

    // Convert from a deforestation raster to vector.
    -var deforestationVector = deforestation.reduceToVectors({
    -   scale: deforestation.projection().nominalScale(),
    -   geometry: protectedArea.geometry(),
    -   labelProperty: ‘lossyear’, // Label polygons with a change year.   maxPixels: 1e13
    -});

    -

    // Count the number of individual change events
    -print(‘Number of change events:’, deforestationVector.size());

    -

    // Display deforestation polygons. Color outline by change year.
    -var deforestationVectorOutline = ee.Image().byte().paint({
    -   featureCollection: deforestationVector,
    -   color: ‘lossyear’,
    -   width: 1
    -});

    -

    Map.addLayer(deforestationVectorOutline, {
    -   palette: [‘yellow’, ‘orange’, ‘red’],
    -   min: 1,
    -   max: 20}, ‘Deforestation vector’);

    +
    +
    +

    +

    Fig. F5.1.5 View of the La Paya protected area in the Colombian Amazon (in white), and deforestation over the period 2001–2020 (in yellows and reds, with darker colors indicating more recent changes)

    +
    +
    +

    We can use Earth Engine to convert the deforestation raster to a set of polygons. The deforestation data are appropriate for this transformation as each deforestation event is labeled categorically by year, and change events are spatially contiguous. This is performed in Earth Engine using the ee.Image.reduceToVectors method, as described earlier in this section.

    +
    // Convert from a deforestation raster to vector.  
+var deforestationVector = deforestation.reduceToVectors({  
+   scale: deforestation.projection().nominalScale(),  
+   geometry: protectedArea.geometry(),  
+   labelProperty: 'lossyear', // Label polygons with a change year.   maxPixels: 1e13  
+});  
+  
+// Count the number of individual change events  
+print('Number of change events:', deforestationVector.size());  
+  
+// Display deforestation polygons. Color outline by change year.  
+var deforestationVectorOutline = ee.Image().byte().paint({  
+   featureCollection: deforestationVector,  
+   color: 'lossyear',  
+   width: 1  
+});  
+  
+Map.addLayer(deforestationVectorOutline, {  
+   palette: ['yellow', 'orange', 'red'],  
+   min: 1,  
+   max: 20}, 'Deforestation vector');

    Fig. F5.1.6 shows a comparison of the raster versus vector representations of deforestation within the protected area.

    -

    -

    Fig. F5.1.6 Raster (left) versus vector (right) representations of deforestation data of the La Paya protected area

    +
    +
    +

    +

    Fig. F5.1.6 Raster (left) versus vector (right) representations of deforestation data of the La Paya protected area

    +
    +

    Having converted from raster to vector, a new set of operations becomes available for post-processing the deforestation data. We might, for instance, be interested in the number of individual change events each year (Fig. F5.1.7):

    -

    var chart = ui.Chart.feature
    -   .histogram({
    -       features: deforestationVector,
    -       property: ‘lossyear’   })
    -   .setOptions({
    -       hAxis: {
    -           title: ‘Year’       },
    -       vAxis: {
    -           title: ‘Number of deforestation events’       },
    -       legend: {
    -           position: ‘none’       }
    -   });print(chart);

    -

    -

    Fig. F5.1.7 Plot of the number of deforestation events in La Paya for the years 2001–2020

    +

    var chart = ui.Chart.feature
    +.histogram({
    +features: deforestationVector,
    +property: ‘lossyear’ })
    +.setOptions({
    +hAxis: {
    +title: ‘Year’ },
    +vAxis: {
    +title: ‘Number of deforestation events’ },
    +legend: {
    +position: ‘none’ }
    +});print(chart);

    +
    +
    +

    +

    Fig. F5.1.7 Plot of the number of deforestation events in La Paya for the years 2001–2020

    +
    +

    There might also be interest in generating point locations for individual change events (e.g., to aid a field campaign):

    -

    // Generate deforestation point locations.
    -var deforestationCentroids = deforestationVector.map(function(feat) {   return feat.centroid();
    -});

    -

    Map.addLayer(deforestationCentroids, {
    -   color: ‘darkblue’}, ‘Deforestation centroids’);

    +
    // Generate deforestation point locations.  
+var deforestationCentroids = deforestationVector.map(function(feat) {   return feat.centroid();  
+});  
+  
+Map.addLayer(deforestationCentroids, {  
+   color: 'darkblue'}, 'Deforestation centroids');

    The vector format allows for easy filtering to only deforestation events of interest, such as only the largest deforestation events:

    -

    // Add a new property to the deforestation FeatureCollection
    -// describing the area of the change polygon.
    -deforestationVector = deforestationVector.map(function(feat) {   return feat.set(‘area’, feat.geometry().area({
    -       maxError: 10   }).divide(10000)); // Convert m^2 to hectare.
    -});

    -

    // Filter the deforestation FeatureCollection for only large-scale (>10 ha) changes
    -var deforestationLarge = deforestationVector.filter(ee.Filter.gt(   ‘area’, 10));

    -

    // Display deforestation area outline by year.
    -var deforestationLargeOutline = ee.Image().byte().paint({
    -   featureCollection: deforestationLarge,
    -   color: ‘lossyear’,
    -   width: 1
    -});

    -

    Map.addLayer(deforestationLargeOutline, {
    -   palette: [‘yellow’, ‘orange’, ‘red’],
    -   min: 1,
    -   max: 20}, ‘Deforestation (>10 ha)’);

    +
    // Add a new property to the deforestation FeatureCollection  
+// describing the area of the change polygon.  
+deforestationVector = deforestationVector.map(function(feat) {   return feat.set('area', feat.geometry().area({  
+       maxError: 10   }).divide(10000)); // Convert m^2 to hectare.  
+});  
+  
+// Filter the deforestation FeatureCollection for only large-scale (>10 ha) changes  
+var deforestationLarge = deforestationVector.filter(ee.Filter.gt(   'area', 10));  
+  
+// Display deforestation area outline by year.  
+var deforestationLargeOutline = ee.Image().byte().paint({  
+   featureCollection: deforestationLarge,  
+   color: 'lossyear',  
+   width: 1  
+});  
+  
+Map.addLayer(deforestationLargeOutline, {  
+   palette: ['yellow', 'orange', 'red'],  
+   min: 1,  
+   max: 20}, 'Deforestation (>10 ha)');
    @@ -926,7 +1065,7 @@ Note
    -

    Code Checkpoint F51b. The book’s repository contains a script that shows what your code should look like at this point.

    +

    Code Checkpoint F51b. The book’s repository contains a script that shows what your code should look like at this point.

    @@ -934,50 +1073,62 @@ Note

    8.1.3 Raster Properties to Vector Fields

    Sometimes we want to extract information from a raster to be included in an existing vector dataset. An example might be estimating a deforestation rate for a set of protected areas. Rather than perform this task on a case-by-case basis, we can attach information generated from an image as a property of a feature.

    The following script shows how this can be used to quantify a deforestation rate for a set of protected areas in the Colombian Amazon.

    -

    // Load required datasets.
    -var gfc = ee.Image(‘UMD/hansen/global_forest_change_2020_v1_8’);
    -var wdpa = ee.FeatureCollection(‘WCMC/WDPA/current/polygons’);

    -

    // Display deforestation.
    -var deforestation = gfc.select(‘lossyear’);

    -

    Map.addLayer(deforestation, {
    -   min: 1,
    -   max: 20,
    -   palette: [‘yellow’, ‘orange’, ‘red’]
    -}, ‘Deforestation raster’);

    -

    // Select protected areas in the Colombian Amazon.
    -var amazonianProtectedAreas = [   ‘Cordillera de los Picachos’, ‘La Paya’, ‘Nukak’,   ‘Serrania de Chiribiquete’,   ‘Sierra de la Macarena’, ‘Tinigua’
    -];

    -

    var wdpaSubset = wdpa.filter(ee.Filter.inList(‘NAME’,
    -   amazonianProtectedAreas));

    -

    // Display protected areas as an outline.
    -var protectedAreasOutline = ee.Image().byte().paint({
    -   featureCollection: wdpaSubset,
    -   color: 1,
    -   width: 1
    -});

    -

    Map.addLayer(protectedAreasOutline, {
    -   palette: ‘white’}, ‘Amazonian protected areas’);

    -

    // Set up map display.
    -Map.centerObject(wdpaSubset);
    -Map.setOptions(‘SATELLITE’);

    -

    var scale = deforestation.projection().nominalScale();

    -

    // Use ‘reduceRegions’ to sum together pixel areas in each protected area.
    -wdpaSubset = deforestation.gte(1)
    -   .multiply(ee.Image.pixelArea().divide(10000)).reduceRegions({
    -       collection: wdpaSubset,
    -       reducer: ee.Reducer.sum().setOutputs([           ‘deforestation_area’]),
    -       scale: scale
    -   });

    -

    print(wdpaSubset); // Note the new ‘deforestation_area’ property.

    -

    The output of this script is an estimate of deforested area in hectares for each reserve. However, as reserve sizes vary substantially by area, we can normalize by the total area of each reserve to quantify rates of change.

    -

    // Normalize by area.
    -wdpaSubset = wdpaSubset.map(   function(feat) {       return feat.set(‘deforestation_rate’,           ee.Number(feat.get(‘deforestation_area’))
    -           .divide(feat.area().divide(10000)) // m2 to ha           .divide(20) // number of years           .multiply(100)); // to percentage points   });// Print to identify rates of change per protected area.
    -// Which has the fastest rate of loss?
    -print(wdpaSubset.reduceColumns({
    -   reducer: ee.Reducer.toList().repeat(2),
    -   selectors: [‘NAME’, ‘deforestation_rate’]
    -}));

    +
    // Load required datasets.  
+var gfc = ee.Image('UMD/hansen/global_forest_change_2020_v1_8');  
+var wdpa = ee.FeatureCollection('WCMC/WDPA/current/polygons');  
+  
+// Display deforestation.  
+var deforestation = gfc.select('lossyear');  
+  
+Map.addLayer(deforestation, {  
+   min: 1,  
+   max: 20,  
+   palette: ['yellow', 'orange', 'red']  
+}, 'Deforestation raster');  
+  
+// Select protected areas in the Colombian Amazon.  
+var amazonianProtectedAreas = [   'Cordillera de los Picachos', 'La Paya', 'Nukak',   'Serrania de Chiribiquete',   'Sierra de la Macarena', 'Tinigua'  
+];  
+  
+var wdpaSubset = wdpa.filter(ee.Filter.inList('NAME',  
+   amazonianProtectedAreas));  
+  
+// Display protected areas as an outline.  
+var protectedAreasOutline = ee.Image().byte().paint({  
+   featureCollection: wdpaSubset,  
+   color: 1,  
+   width: 1  
+});  
+  
+Map.addLayer(protectedAreasOutline, {  
+   palette: 'white'}, 'Amazonian protected areas');  
+  
+// Set up map display.  
+Map.centerObject(wdpaSubset);  
+Map.setOptions('SATELLITE');  
+  
+var scale = deforestation.projection().nominalScale();  
+  
+// Use 'reduceRegions' to sum together pixel areas in each protected area.  
+wdpaSubset = deforestation.gte(1)  
+   .multiply(ee.Image.pixelArea().divide(10000)).reduceRegions({  
+       collection: wdpaSubset,  
+       reducer: ee.Reducer.sum().setOutputs([           'deforestation_area']),  
+       scale: scale  
+   });  
+  
+print(wdpaSubset); // Note the new 'deforestation_area' property.
+
+The output of this script is an estimate of deforested area in hectares for each reserve. However, as reserve sizes vary substantially by area, we can normalize by the total area of each reserve to quantify rates of change.
+
+// Normalize by area.  
+wdpaSubset = wdpaSubset.map(   function(feat) {       return feat.set('deforestation_rate',           ee.Number(feat.get('deforestation_area'))  
+           .divide(feat.area().divide(10000)) // m2 to ha           .divide(20) // number of years           .multiply(100)); // to percentage points   });// Print to identify rates of change per protected area.  
+// Which has the fastest rate of loss?  
+print(wdpaSubset.reduceColumns({  
+   reducer: ee.Reducer.toList().repeat(2),  
+   selectors: ['NAME', 'deforestation_rate']  
+}));
    @@ -988,53 +1139,61 @@ Note
    -

    Code Checkpoint F51c. The book’s repository contains a script that shows what your code should look like at this point.

    +

    Code Checkpoint F51c. The book’s repository contains a script that shows what your code should look like at this point.

    8.2 Vector-to-Raster Conversion

    -

    In Sect. 1, we used the protected area feature collection as its original vector format. In this section, we will rasterize the protected area polygons to produce a mask and use this to assess rates of forest change.

    +

    In Sect. 1, we used the protected area feature collection as its original vector format. In this section, we will rasterize the protected area polygons to produce a mask and use this to assess rates of forest change.

    8.2.1 Polygons to a Mask

    -

    The most common operation to convert from vector to raster is the production of binary image masks, describing whether a pixel intersects a line or falls within a polygon. To convert from vector to a raster mask, we can use the ee.FeatureCollection.reduceToImage method. Let’s continue with our example of the WDPA database and Global Forest Change data from the previous section:

    -

    // Load required datasets.
    -var gfc = ee.Image(‘UMD/hansen/global_forest_change_2020_v1_8’);
    -var wdpa = ee.FeatureCollection(‘WCMC/WDPA/current/polygons’);

    -

    // Get deforestation.
    -var deforestation = gfc.select(‘lossyear’);

    -

    // Generate a new property called ‘protected’ to apply to the output mask.
    -var wdpa = wdpa.map(function(feat) {   return feat.set(‘protected’, 1);
    -});

    -

    // Rasterize using the new property.
    -// unmask() sets areas outside protected area polygons to 0.
    -var wdpaMask = wdpa.reduceToImage([‘protected’], ee.Reducer.first())
    -   .unmask();

    -

    // Center on Colombia.
    -Map.setCenter(-75, 3, 6);

    -

    // Display on map.
    -Map.addLayer(wdpaMask, {
    -   min: 0,
    -   max: 1}, ‘Protected areas (mask)’);

    +

    The most common operation to convert from vector to raster is the production of binary image masks, describing whether a pixel intersects a line or falls within a polygon. To convert from vector to a raster mask, we can use the ee.FeatureCollection.reduceToImage method. Let’s continue with our example of the WDPA database and Global Forest Change data from the previous section:

    +
    // Load required datasets.  
+var gfc = ee.Image('UMD/hansen/global_forest_change_2020_v1_8');  
+var wdpa = ee.FeatureCollection('WCMC/WDPA/current/polygons');  
+  
+// Get deforestation.  
+var deforestation = gfc.select('lossyear');  
+  
+// Generate a new property called 'protected' to apply to the output mask.  
+var wdpa = wdpa.map(function(feat) {   return feat.set('protected', 1);  
+});  
+  
+// Rasterize using the new property.  
+// unmask() sets areas outside protected area polygons to 0.  
+var wdpaMask = wdpa.reduceToImage(['protected'], ee.Reducer.first())  
+   .unmask();  
+  
+// Center on Colombia.  
+Map.setCenter(-75, 3, 6);  
+  
+// Display on map.  
+Map.addLayer(wdpaMask, {  
+   min: 0,  
+   max: 1}, 'Protected areas (mask)');

    We can use this mask to, for example, highlight only deforestation that occurs within a protected area using logical operations:

    -

    // Set the deforestation layer to 0 where outside a protected area.
    -var deforestationProtected = deforestation.where(wdpaMask.eq(0), 0);

    -

    // Update mask to hide where deforestation layer = 0
    -var deforestationProtected = deforestationProtected
    -   .updateMask(deforestationProtected.gt(0));

    -

    // Display deforestation in protected areas
    -Map.addLayer(deforestationProtected, {
    -   min: 1,
    -   max: 20,
    -   palette: [‘yellow’, ‘orange’, ‘red’]
    -}, ‘Deforestation protected’);

    -

    In the above example we generated a simple binary mask, but reduceToImage can also preserve a numerical property of the input polygons. For example, we might want to be able to determine which protected area each pixel represents. In this case, we can produce an image with the unique ID of each protected area:

    -

    // Produce an image with unique ID of protected areas.
    -var wdpaId = wdpa.reduceToImage([‘WDPAID’], ee.Reducer.first());

    -

    Map.addLayer(wdpaId, {
    -   min: 1,
    -   max: 100000}, ‘Protected area ID’);

    +
    // Set the deforestation layer to 0 where outside a protected area.  
+var deforestationProtected = deforestation.where(wdpaMask.eq(0), 0);  
+  
+// Update mask to hide where deforestation layer = 0  
+var deforestationProtected = deforestationProtected  
+   .updateMask(deforestationProtected.gt(0));  
+  
+// Display deforestation in protected areas  
+Map.addLayer(deforestationProtected, {  
+   min: 1,  
+   max: 20,  
+   palette: ['yellow', 'orange', 'red']  
+}, 'Deforestation protected');
    +

    In the above example we generated a simple binary mask, but reduceToImage can also preserve a numerical property of the input polygons. For example, we might want to be able to determine which protected area each pixel represents. In this case, we can produce an image with the unique ID of each protected area:

    +
    // Produce an image with unique ID of protected areas.  
+var wdpaId = wdpa.reduceToImage(['WDPAID'], ee.Reducer.first());  
+  
+Map.addLayer(wdpaId, {  
+   min: 1,  
+   max: 100000}, 'Protected area ID');

    This output can be useful when performing large-scale raster operations, such as efficiently calculating deforestation rates for multiple protected areas.

    @@ -1046,86 +1205,102 @@ Note
    -

    Code Checkpoint F51d. The book’s repository contains a script that shows what your code should look like at this point.

    +

    Code Checkpoint F51d. The book’s repository contains a script that shows what your code should look like at this point.

    8.2.2 A More Complex Example

    -

    The reduceToImage method is not the only way to convert a feature collection to an image. We will create a distance image layer from the boundary of the protected area using distance. For this example, we return to the La Paya protected area explored in Sect. 1.

    -

    // Load required datasets.
    -var gfc = ee.Image(‘UMD/hansen/global_forest_change_2020_v1_8’);
    -var wdpa = ee.FeatureCollection(‘WCMC/WDPA/current/polygons’);

    -

    // Select a single protected area.
    -var protectedArea = wdpa.filter(ee.Filter.equals(‘NAME’, ‘La Paya’));

    -

    // Maximum distance in meters is set in the brackets.
    -var distance = protectedArea.distance(1000000);

    -

    Map.addLayer(distance, {
    -   min: 0,
    -   max: 20000,
    -   palette: [‘white’, ‘grey’, ‘black’],
    -   opacity: 0.6}, ‘Distance’);

    -

    Map.centerObject(protectedArea);

    +

    The reduceToImage method is not the only way to convert a feature collection to an image. We will create a distance image layer from the boundary of the protected area using distance. For this example, we return to the La Paya protected area explored in Sect. 1.

    +
    // Load required datasets.  
+var gfc = ee.Image('UMD/hansen/global_forest_change_2020_v1_8');  
+var wdpa = ee.FeatureCollection('WCMC/WDPA/current/polygons');  
+  
+// Select a single protected area.  
+var protectedArea = wdpa.filter(ee.Filter.equals('NAME', 'La Paya'));  
+  
+// Maximum distance in meters is set in the brackets.  
+var distance = protectedArea.distance(1000000);  
+  
+Map.addLayer(distance, {  
+   min: 0,  
+   max: 20000,  
+   palette: ['white', 'grey', 'black'],  
+   opacity: 0.6}, 'Distance');  
+  
+Map.centerObject(protectedArea);

    We can also show the distance inside and outside of the boundary by using the rasterized protected area (Fig. F5.1.8).

    -

    // Produce a raster of inside/outside the protected area.
    -var protectedAreaRaster = protectedArea.map(function(feat) {   return feat.set(‘protected’, 1);
    -}).reduceToImage([‘protected’], ee.Reducer.first());

    -

    Map.addLayer(distance.updateMask(protectedAreaRaster), {
    -   min: 0,
    -   max: 20000}, ‘Distance inside protected area’);

    -

    Map.addLayer(distance.updateMask(protectedAreaRaster.unmask()
    -.not()), {
    -   min: 0,
    -   max: 20000}, ‘Distance outside protected area’);

    +
    // Produce a raster of inside/outside the protected area.  
+var protectedAreaRaster = protectedArea.map(function(feat) {   return feat.set('protected', 1);  
+}).reduceToImage(['protected'], ee.Reducer.first());  
+  
+Map.addLayer(distance.updateMask(protectedAreaRaster), {  
+   min: 0,  
+   max: 20000}, 'Distance inside protected area');  
+  
+Map.addLayer(distance.updateMask(protectedAreaRaster.unmask()  
+.not()), {  
+   min: 0,  
+   max: 20000}, 'Distance outside protected area');

    -

    -

    Fig. F5.1.8 Distance from the La Paya boundary (left), distance within the La Paya (middle), and distance outside the La Paya (right)

    +
    +
    +

    +

    Fig. F5.1.8 Distance from the La Paya boundary (left), distance within the La Paya (middle), and distance outside the La Paya (right)

    +
    +

    Sometimes it makes sense to work with objects in raster imagery. This is an unusual case of vector-like operations conducted with raster data. There is a good reason for this where the vector equivalent would be computationally burdensome.

    An example of this is estimating deforestation rates by distance to the edge of the protected area, as it is common that rates of change will be higher at the boundary of a protected area. We will create a distance raster with three zones from the La Paya boundary (>1 km, >2 km, >3 km, and >4 km) and to estimate the deforestation by distance from the boundary (Fig. F5.1.9).

    -

    var distanceZones = ee.Image(0)
    -   .where(distance.gt(0), 1)
    -   .where(distance.gt(1000), 2)
    -   .where(distance.gt(3000), 3)
    -   .updateMask(distance.lte(5000));

    +

    var distanceZones = ee.Image(0)
    +.where(distance.gt(0), 1)
    +.where(distance.gt(1000), 2)
    +.where(distance.gt(3000), 3)
    +.updateMask(distance.lte(5000));

    Map.addLayer(distanceZones, {}, ‘Distance zones’);

    -

    var deforestation = gfc.select(‘loss’);
    -var deforestation1km = deforestation.updateMask(distanceZones.eq(1));
    -var deforestation3km = deforestation.updateMask(distanceZones.lte(2));
    -var deforestation5km = deforestation.updateMask(distanceZones.lte(3));

    +

    var deforestation = gfc.select(‘loss’);
    +var deforestation1km = deforestation.updateMask(distanceZones.eq(1));
    +var deforestation3km = deforestation.updateMask(distanceZones.lte(2));
    +var deforestation5km = deforestation.updateMask(distanceZones.lte(3));

    Map.addLayer(deforestation1km, {
    -   min: 0,
    -   max: 1}, ‘Deforestation within a 1km buffer’);
    +min: 0,
    +max: 1}, ‘Deforestation within a 1km buffer’);
    Map.addLayer(deforestation3km, {
    -   min: 0,
    -   max: 1,
    -   opacity: 0.5}, ‘Deforestation within a 3km buffer’);
    +min: 0,
    +max: 1,
    +opacity: 0.5}, ‘Deforestation within a 3km buffer’);
    Map.addLayer(deforestation5km, {
    -   min: 0,
    -   max: 1,
    -   opacity: 0.5}, ‘Deforestation within a 5km buffer’);

    +min: 0,
    +max: 1,
    +opacity: 0.5}, ‘Deforestation within a 5km buffer’);

    -

    -

    Fig. F5.1.9 Distance zones (top left) and deforestation by zone (<1 km, <3 km, and <5 km)

    +
    +
    +

    +

    Fig. F5.1.9 Distance zones (top left) and deforestation by zone (<1 km, <3 km, and <5 km)

    +
    +

    Lastly, we can estimate the deforestation area within 1 km of the protected area but only outside of the boundary.

    -

    var deforestation1kmOutside = deforestation1km
    -   .updateMask(protectedAreaRaster.unmask().not());

    -

    // Get the value of each pixel in square meters
    -// and divide by 10000 to convert to hectares.
    -var deforestation1kmOutsideArea = deforestation1kmOutside.eq(1)
    -   .multiply(ee.Image.pixelArea()).divide(10000);

    -

    // We need to set a larger geometry than the protected area
    -// for the geometry parameter in reduceRegion().
    -var deforestationEstimate = deforestation1kmOutsideArea
    -   .reduceRegion({
    -       reducer: ee.Reducer.sum(),
    -       geometry: protectedArea.geometry().buffer(1000),
    -       scale: deforestation.projection().nominalScale()
    -   });

    -

    print(‘Deforestation within a 1km buffer outside the protected area (ha)’,
    -   deforestationEstimate);

    +

    var deforestation1kmOutside = deforestation1km
    +.updateMask(protectedAreaRaster.unmask().not());

    +
    // Get the value of each pixel in square meters  
+// and divide by 10000 to convert to hectares.  
+var deforestation1kmOutsideArea = deforestation1kmOutside.eq(1)  
+   .multiply(ee.Image.pixelArea()).divide(10000);  
+  
+// We need to set a larger geometry than the protected area  
+// for the geometry parameter in reduceRegion().  
+var deforestationEstimate = deforestation1kmOutsideArea  
+   .reduceRegion({  
+       reducer: ee.Reducer.sum(),  
+       geometry: protectedArea.geometry().buffer(1000),  
+       scale: deforestation.projection().nominalScale()  
+   });  
+  
+print('Deforestation within a 1km buffer outside the protected area (ha)',  
+   deforestationEstimate);
    @@ -1136,14 +1311,14 @@ Note
    -

    Code Checkpoint F51e. The book’s repository contains a script that shows what your code should look like at this point.

    +

    Code Checkpoint F51e. The book’s repository contains a script that shows what your code should look like at this point.

    Synthesis

    -

    Question 1. In this lab, we quantified rates of deforestation in La Paya. There is another protected area in the Colombian Amazon named Tinigua. By modifying the existing scripts, determine how the dynamics of forest change in Tinigua compare to those in La Paya with respect to:

    +

    Question 1. In this lab, we quantified rates of deforestation in La Paya. There is another protected area in the Colombian Amazon named Tinigua. By modifying the existing scripts, determine how the dynamics of forest change in Tinigua compare to those in La Paya with respect to:

    • the number of deforestation events
    • the year with the greatest number of change events
      • @@ -1152,7 +1327,7 @@ Note

    Question 2. In Sect. 1.4, we only considered losses of tree cover, but many protected areas will also have increases in tree cover from regrowth (which is typical of shifting agriculture). Calculate growth in hectares using the Global Forest Change dataset’s gain layer for the six protected areas in Sect. 1.4 by extracting the raster properties and adding them to vector fields. Which has the greatest area of regrowth? Is this likely to be sufficient to balance out the rates of forest loss? Note: The gain layer shows locations where tree cover has increased for the period 2001–2012 (0 = no gain, 1 = tree cover increase), so for comparability use deforestation between the same time period of 2001–2012.

    Question 3. In Sect. 2.2, we considered rates of deforestation in a buffer zone around La Paya. Estimate the deforestation rates inside of La Paya using buffer zones. Is forest loss more common close to the boundary of the reserve?

    -

    Question 4. Sometimes it’s advantageous to perform processing using raster operations, particularly at large scales. It is possible to perform many of the tasks in Sect. 1.3 and 1.4 by first converting the protected area vector to raster, and then using only raster operations. As an example, can you display only deforestation events >10 ha in La Paya using only raster data? (Hint: Consider using ee.Image.connectedPixelCount. You may also want to also look at Sect. 2.1).

    +

    Question 4. Sometimes it’s advantageous to perform processing using raster operations, particularly at large scales. It is possible to perform many of the tasks in Sect. 1.3 and 1.4 by first converting the protected area vector to raster, and then using only raster operations. As an example, can you display only deforestation events >10 ha in La Paya using only raster data? (Hint: Consider using ee.Image.connectedPixelCount. You may also want to also look at Sect. 2.1).

    Conclusion

    @@ -1161,10 +1336,9 @@ Note

    9 Zonal Statistics

    -

    ::: {.callout-tip} # Chapter Information

    +

    :::{.callout-tip} # Chapter Information

    Author

    -

     

    Sara Winsemius and Justin Braaten

    @@ -1173,7 +1347,6 @@ Note

    Learning Outcomes

    -

     

    • Buffering points as square or circular regions.
    • Writing and applying functions with optional parameters.
      • @@ -1186,19 +1359,19 @@ Note

      Assumes you know how to:

      • Recognize similarities and differences among Landsat 5, 7, and 8 spectral bands (Part F1, Part F2, Part F3).
        • -
      • Understand distinctions among Image, ImageCollection, Feature and FeatureCollection Earth Engine objects (Part F1, Part F2, Part F5).
        • +
      • Understand distinctions among Image, ImageCollection, Feature and FeatureCollection Earth Engine objects (Part F1, Part F2, Part F5).
      • Use drawing tools to create points, lines, and polygons (Chap. F2.1).
        • -
      • Write a function and map it over an ImageCollection (Chap. F4.0).
        • +
      • Write a function and map it over an ImageCollection (Chap. F4.0).
      • Mask cloud, cloud shadow, snow/ice, and other undesired pixels (Chap. F4.3).
      • Export calculated data to tables with Tasks (Chap. F5.0).
        • -
      • Understand the differences between raster and vector data (Chap. F5.0, Chap. F5.1).
        • -
      • Write a function and map it over a FeatureCollection (Chap. F5.1).
        • +
      • Understand the differences between raster and vector data (Chap. F5.0, Chap. F5.1).
        • +
      • Write a function and map it over a FeatureCollection (Chap. F5.1).
    -

    9.1 Introduction to Theory 

    -

    Anyone working with field data collected at plots will likely need to summarize raster-based data associated with those plots. For instance, they need to know the Normalized Difference Vegetation Index (NDVI), precipitation, or elevation for each plot (or surrounding region). Calculating statistics from a raster within given regions is called zonal statistics. Zonal statistics were calculated in Chaps. F5.0 and F5.1 using ee.Image.ReduceRegions. Here, we present a more general approach to calculating zonal statistics with a custom function that works for both ee.Image and ee.ImageCollection objects. In addition to its flexibility, the reduction method used here is less prone to “Computed value is too large” errors that can occur when using ReduceRegions with very large or complex ee.FeatureCollection object inputs.

    -

    The zonal statistics function in this chapter works for an Image or an ImageCollection. Running the function over an ImageCollection will produce a table with values from each image in the collection per point. Image collections can be processed before extraction as needed—for example, by masking clouds from satellite imagery or by constraining the dates needed for a particular research question. In this tutorial, the data extracted from rasters are exported to a table for analysis, where each row of the table corresponds to a unique point-image combination.

    +

    9.1 Introduction to Theory

    +

    Anyone working with field data collected at plots will likely need to summarize raster-based data associated with those plots. For instance, they need to know the Normalized Difference Vegetation Index (NDVI), precipitation, or elevation for each plot (or surrounding region). Calculating statistics from a raster within given regions is called zonal statistics. Zonal statistics were calculated in Chaps. F5.0 and F5.1 using ee.Image.ReduceRegions. Here, we present a more general approach to calculating zonal statistics with a custom function that works for both ee.Image and ee.ImageCollection objects. In addition to its flexibility, the reduction method used here is less prone to “Computed value is too large” errors that can occur when using ReduceRegions with very large or complex ee.FeatureCollection object inputs.

    +

    The zonal statistics function in this chapter works for an Image or an ImageCollection. Running the function over an ImageCollection will produce a table with values from each image in the collection per point. Image collections can be processed before extraction as needed—for example, by masking clouds from satellite imagery or by constraining the dates needed for a particular research question. In this tutorial, the data extracted from rasters are exported to a table for analysis, where each row of the table corresponds to a unique point-image combination.

    In fieldwork, researchers often work with plots, which are commonly recorded as polygon files or as a center point with a set radius. It is rare that plots will be set directly in the center of pixels from your desired raster dataset, and many field GPS units have positioning errors. Because of these issues, it may be important to use a statistic of adjacent pixels (as described in Chap. F3.2) to estimate the central value in what’s often called a neighborhood mean or focal mean (Cansler and McKenzie 2012, Miller and Thode 2007).

    To choose the size of your neighborhood, you will need to consider your research questions, the spatial resolution of the dataset, the size of your field plot, and the error from your GPS. For example, the raster value extracted for randomly placed 20 m diameter plots would likely merit use of a neighborhood mean when using Sentinel-2 or Landsat 8—at 10 m and 30 m spatial resolution, respectively—while using a thermal band from MODIS (Moderate Resolution Imaging Spectroradiometer) at 1000 m may not. While much of this tutorial is written with plot points and buffers in mind, a polygon asset with predefined regions will serve the same purpose.

    @@ -1212,7 +1385,7 @@ Note

    9.2.1 Function: bufferPoints(radius, bounds)

    Our first function, bufferPoints, returns a function for adding a buffer to points and optionally transforming to rectangular bounds (see Table F5.2.1).

    -

    Table F5.2.1 Parameters for bufferPoints

    +

    Table F5.2.1 Parameters for bufferPoints

    Parameter

    Type

    Description

    @@ -1222,17 +1395,17 @@ Note

    [bounds=false]

    Boolean

    An optional flag indicating whether to transform buffered point (i.e., a circle) to square bounds.

    -

    function bufferPoints(radius, bounds) {   return function(pt) {
    -       pt = ee.Feature(pt);       return bounds ? pt.buffer(radius).bounds() : pt.buffer(
    -           radius);
    -   };
    +

    function bufferPoints(radius, bounds) { return function(pt) {
    +pt = ee.Feature(pt); return bounds ? pt.buffer(radius).bounds() : pt.buffer(
    +radius);
    +};
    }

    9.2.2 Function: zonalStats(fc, params)

    -

    The second function, zonalStats, reduces images in an ImageCollection by regions defined in a FeatureCollection. Note that reductions can return null statistics that you might want to filter out of the resulting feature collection. Null statistics occur when there are no valid pixels intersecting the region being reduced. This situation can be caused by points that are outside of an image or in regions that are masked for quality or clouds.

    +

    The second function, zonalStats, reduces images in an ImageCollection by regions defined in a FeatureCollection. Note that reductions can return null statistics that you might want to filter out of the resulting feature collection. Null statistics occur when there are no valid pixels intersecting the region being reduced. This situation can be caused by points that are outside of an image or in regions that are masked for quality or clouds.

    This function is written to include many optional parameters (see Table F5.2.2). Look at the function carefully and note how it is written to include defaults that make it easy to apply the basic function while allowing customization.

    -

    Table F5.2.2 Parameters for zonalStats

    +

    Table F5.2.2 Parameters for zonalStats

    Parameter

    Type

    Description

    @@ -1271,41 +1444,41 @@ Note

    The desired name of the datetime field. The datetime refers to the ‘system:time_start’ value of the ee.Image being reduced. Optional.

    [params.datetimeFormat=’YYYY-MM-dd HH:mm:ss]

    String

    -

    The desired datetime format. Use ISO 8601 data string standards. The datetime string is derived from the ‘system:time_start’ value of the ee.Image being reduced. Optional.

    -

    function zonalStats(ic, fc, params) {   // Initialize internal params dictionary.   var _params = {
    -       reducer: ee.Reducer.mean(),
    -       scale: null,
    -       crs: null,
    -       bands: null,
    -       bandsRename: null,
    -       imgProps: null,
    -       imgPropsRename: null,
    -       datetimeName: ‘datetime’,
    -       datetimeFormat: ‘YYYY-MM-dd HH:mm:ss’   };   // Replace initialized params with provided params.   if (params) {       for (var param in params) {
    -           _params[param] = params[param] || _params[param];
    -       }
    -   }   // Set default parameters based on an image representative.   var imgRep = ic.first();   var nonSystemImgProps = ee.Feature(null)
    -       .copyProperties(imgRep).propertyNames();   if (!_params.bands) _params.bands = imgRep.bandNames();   if (!_params.bandsRename) _params.bandsRename = _params.bands;   if (!_params.imgProps) _params.imgProps = nonSystemImgProps;   if (!_params.imgPropsRename) _params.imgPropsRename = _params
    -       .imgProps;   // Map the reduceRegions function over the image collection.   var results = ic.map(function(img) {       // Select bands (optionally rename), set a datetime & timestamp property.       img = ee.Image(img.select(_params.bands, _params
    -               .bandsRename))           // Add datetime and timestamp features.           .set(_params.datetimeName, img.date().format(
    -               _params.datetimeFormat))           .set(‘timestamp’, img.get(‘system:time_start’));       // Define final image property dictionary to set in output features.       var propsFrom = ee.List(_params.imgProps)           .cat(ee.List([_params.datetimeName,           ‘timestamp’]));       var propsTo = ee.List(_params.imgPropsRename)           .cat(ee.List([_params.datetimeName,           ‘timestamp’]));       var imgProps = img.toDictionary(propsFrom).rename(
    -           propsFrom, propsTo);       // Subset points that intersect the given image.       var fcSub = fc.filterBounds(img.geometry());       // Reduce the image by regions.       return img.reduceRegions({
    -               collection: fcSub,
    -               reducer: _params.reducer,               scale: _params.scale,               crs: _params.crs
    -           })           // Add metadata to each feature.           .map(function(f) {               return f.set(imgProps);
    -           });       // Converts the feature collection of feature collections to a single       //feature collection.   }).flatten();   return results;
    +

    The desired datetime format. Use ISO 8601 data string standards. The datetime string is derived from the ‘system:time_start’ value of the ee.Image being reduced. Optional.

    +

    function zonalStats(ic, fc, params) { // Initialize internal params dictionary. var _params = {
    +reducer: ee.Reducer.mean(),
    +scale: null,
    +crs: null,
    +bands: null,
    +bandsRename: null,
    +imgProps: null,
    +imgPropsRename: null,
    +datetimeName: ‘datetime’,
    +datetimeFormat: ‘YYYY-MM-dd HH:mm:ss’ }; // Replace initialized params with provided params. if (params) { for (var param in params) {
    +_params[param] = params[param] || _params[param];
    +}
    +} // Set default parameters based on an image representative. var imgRep = ic.first(); var nonSystemImgProps = ee.Feature(null)
    +.copyProperties(imgRep).propertyNames(); if (!_params.bands) _params.bands = imgRep.bandNames(); if (!_params.bandsRename) _params.bandsRename = _params.bands; if (!_params.imgProps) _params.imgProps = nonSystemImgProps; if (!_params.imgPropsRename) _params.imgPropsRename = _params
    +.imgProps; // Map the reduceRegions function over the image collection. var results = ic.map(function(img) { // Select bands (optionally rename), set a datetime & timestamp property. img = ee.Image(img.select(_params.bands, _params
    +.bandsRename)) // Add datetime and timestamp features. .set(_params.datetimeName, img.date().format(
    +_params.datetimeFormat)) .set(‘timestamp’, img.get(‘system:time_start’)); // Define final image property dictionary to set in output features. var propsFrom = ee.List(_params.imgProps) .cat(ee.List([_params.datetimeName, ‘timestamp’])); var propsTo = ee.List(_params.imgPropsRename) .cat(ee.List([_params.datetimeName, ‘timestamp’])); var imgProps = img.toDictionary(propsFrom).rename(
    +propsFrom, propsTo); // Subset points that intersect the given image. var fcSub = fc.filterBounds(img.geometry()); // Reduce the image by regions. return img.reduceRegions({
    +collection: fcSub,
    +reducer: _params.reducer, scale: _params.scale, crs: _params.crs
    +}) // Add metadata to each feature. .map(function(f) { return f.set(imgProps);
    +}); // Converts the feature collection of feature collections to a single //feature collection. }).flatten(); return results;
    }

    9.3 Point Collection Creation

    -

    Below, we create a set of points that form the basis of the zonal statistics calculations. Note that a unique plot_id property is added to each point. A unique plot or point ID is important to include in your vector dataset for future filtering and joining.

    -

    var pts = ee.FeatureCollection([   ee.Feature(ee.Geometry.Point([-118.6010, 37.0777]), {
    -       plot_id: 1   }),   ee.Feature(ee.Geometry.Point([-118.5896, 37.0778]), {
    -       plot_id: 2   }),   ee.Feature(ee.Geometry.Point([-118.5842, 37.0805]), {
    -       plot_id: 3   }),   ee.Feature(ee.Geometry.Point([-118.5994, 37.0936]), {
    -       plot_id: 4   }),   ee.Feature(ee.Geometry.Point([-118.5861, 37.0567]), {
    -       plot_id: 5   })
    +

    Below, we create a set of points that form the basis of the zonal statistics calculations. Note that a unique plot_id property is added to each point. A unique plot or point ID is important to include in your vector dataset for future filtering and joining.

    +

    var pts = ee.FeatureCollection([ ee.Feature(ee.Geometry.Point([-118.6010, 37.0777]), {
    +plot_id: 1 }), ee.Feature(ee.Geometry.Point([-118.5896, 37.0778]), {
    +plot_id: 2 }), ee.Feature(ee.Geometry.Point([-118.5842, 37.0805]), {
    +plot_id: 3 }), ee.Feature(ee.Geometry.Point([-118.5994, 37.0936]), {
    +plot_id: 4 }), ee.Feature(ee.Geometry.Point([-118.5861, 37.0567]), {
    +plot_id: 5 })
    ]);print(‘Points of interest’, pts);

    @@ -1317,7 +1490,7 @@ Note
    -

    Code Checkpoint F52a. The book’s repository contains a script that shows what your code should look like at this point.

    +

    Code Checkpoint F52a. The book’s repository contains a script that shows what your code should look like at this point.

    @@ -1334,52 +1507,65 @@ Note

    9.4.1 Topographic Variables

    This example demonstrates how to calculate zonal statistics for a single multiband image. This Digital Elevation Model (DEM) contains a single topographic band representing elevation.

    ###Buffer the Points

    -

    Nex, we will apply a 45 m radius buffer to the points defined previously by mapping the bufferPoints function over the feature collection. The radius is set to 45 m to correspond to the 90 m pixel resolution of the DEM. In this case, circles are used instead of squares (set the second argument as false, i.e., do not use bounds).

    -

    // Buffer the points.
    -var ptsTopo = pts.map(bufferPoints(45, false));

    +

    Nex, we will apply a 45 m radius buffer to the points defined previously by mapping the bufferPoints function over the feature collection. The radius is set to 45 m to correspond to the 90 m pixel resolution of the DEM. In this case, circles are used instead of squares (set the second argument as false, i.e., do not use bounds).

    +
    // Buffer the points.  
+var ptsTopo = pts.map(bufferPoints(45, false));

    ###Calculate Zonal Statistics

    -

    There are two important things to note about the zonalStats function that this example addresses:

    +

    There are two important things to note about the zonalStats function that this example addresses:

    • It accepts only an ee.ImageCollection, not an ee.Image; single images must be wrapped in an ImageCollection.
      • -
    • It expects every image in the input image collection to have a timestamp property named ‘system:time_start’ with values representing milliseconds from 00:00:00 UTC on 1 January 1970. Most datasets should have this property, if not, one should be added.
      • +
    • It expects every image in the input image collection to have a timestamp property named ‘system:time_start’ with values representing milliseconds from 00:00:00 UTC on 1 January 1970. Most datasets should have this property, if not, one should be added.
    -

    // Import the MERIT global elevation dataset.
    -var elev = ee.Image(‘MERIT/DEM/v1_0_3’);

    -

    // Calculate slope from the DEM.
    -var slope = ee.Terrain.slope(elev);

    -

    // Concatenate elevation and slope as two bands of an image.
    -var topo = ee.Image.cat(elev, slope)
    -    // Computed images do not have a ‘system:time_start’ property; add one based
    -    // on when the data were collected.   .set(‘system:time_start’, ee.Date(‘2000-01-01’).millis());

    -

    // Wrap the single image in an ImageCollection for use in the
    -// zonalStats function.
    -var topoCol = ee.ImageCollection([topo]);

    -

    Define arguments for the zonalStats function and then run it. Note that we are accepting defaults for the reducer, scale, Coordinate Reference System (CRS), and image properties to copy over to the resulting feature collection. Refer to the function definition above for defaults.

    -

    // Define parameters for the zonalStats function.
    -var params = {
    -   bands: [0, 1],
    -   bandsRename: [‘elevation’, ‘slope’]
    -};

    -

    // Extract zonal statistics per point per image.
    -var ptsTopoStats = zonalStats(topoCol, ptsTopo, params);print(‘Topo zonal stats table’, ptsTopoStats);

    -

    // Display the layers on the map.
    -Map.setCenter(-118.5957, 37.0775, 13);
    -Map.addLayer(topoCol.select(0), {
    -   min: 2400,
    -   max: 4200}, ‘Elevation’);
    -Map.addLayer(topoCol.select(1), {
    -   min: 0,
    -   max: 60}, ‘Slope’);
    -Map.addLayer(pts, {
    -   color: ‘purple’}, ‘Points’);
    -Map.addLayer(ptsTopo, {
    -   color: ‘yellow’}, ‘Points w/ buffer’);

    -

    The result is a copy of the buffered point feature collection with new properties added for the region reduction of each selected image band according to the given reducer. A part of the FeatureCollection is shown in Fig. F5.2.1. The data in that FeatureCollection corresponds to a table containing the information of Table F5.2.3. See Fig. F5.2.2 for a graphical representation of the points and the topographic data being summarized.

    -

    -

    Fig. F5.2.1 A part of the FeatureCollection produced by calculating the zonal statistics

    -

    -

    Fig. F5.2.2 Sample points and topographic slope. Elevation and slope values for regions intersecting each buffered point are reduced and attached as properties of the points.

    -

    Table F5.2.3 Example output from zonalStats organized as a table. Rows correspond to collection features and columns are feature properties. Note that elevation and slope values in this table are rounded to the nearest tenth for brevity.

    +
    // Import the MERIT global elevation dataset.  
+var elev = ee.Image('MERIT/DEM/v1_0_3');  
+  
+// Calculate slope from the DEM.  
+var slope = ee.Terrain.slope(elev);  
+  
+// Concatenate elevation and slope as two bands of an image.  
+var topo = ee.Image.cat(elev, slope)  
+    // Computed images do not have a 'system:time_start' property; add one based  
+    // on when the data were collected.   .set('system:time_start', ee.Date('2000-01-01').millis());  
+  
+// Wrap the single image in an ImageCollection for use in the  
+// zonalStats function.  
+var topoCol = ee.ImageCollection([topo]);
    +

    Define arguments for the zonalStats function and then run it. Note that we are accepting defaults for the reducer, scale, Coordinate Reference System (CRS), and image properties to copy over to the resulting feature collection. Refer to the function definition above for defaults.

    +
    // Define parameters for the zonalStats function.  
+var params = {  
+   bands: [0, 1],  
+   bandsRename: ['elevation', 'slope']  
+};  
+  
+// Extract zonal statistics per point per image.  
+var ptsTopoStats = zonalStats(topoCol, ptsTopo, params);print('Topo zonal stats table', ptsTopoStats);  
+  
+// Display the layers on the map.  
+Map.setCenter(-118.5957, 37.0775, 13);  
+Map.addLayer(topoCol.select(0), {  
+   min: 2400,  
+   max: 4200}, 'Elevation');  
+Map.addLayer(topoCol.select(1), {  
+   min: 0,  
+   max: 60}, 'Slope');  
+Map.addLayer(pts, {  
+   color: 'purple'}, 'Points');  
+Map.addLayer(ptsTopo, {  
+   color: 'yellow'}, 'Points w/ buffer');
    +

    The result is a copy of the buffered point feature collection with new properties added for the region reduction of each selected image band according to the given reducer. A part of the FeatureCollection is shown in Fig. F5.2.1. The data in that FeatureCollection corresponds to a table containing the information of Table F5.2.3. See Fig. F5.2.2 for a graphical representation of the points and the topographic data being summarized.

    +
    +
    +

    +

    Fig. F5.2.1 A part of the FeatureCollection produced by calculating the zonal statistics

    +
    +
    +
    +
    +

    +

    Fig. F5.2.2 Sample points and topographic slope. Elevation and slope values for regions intersecting each buffered point are reduced and attached as properties of the points.

    +
    +
    +

    Table F5.2.3 Example output from zonalStats organized as a table. Rows correspond to collection features and columns are feature properties. Note that elevation and slope values in this table are rounded to the nearest tenth for brevity.

    plot_id

    timestamp

    datetime

    @@ -1413,108 +1599,120 @@ Map.addLayer(ptsTopo, {

    9.4.2 MODIS Time Series

    -

    A time series of MODIS eight-day surface reflectance composites demonstrates how to calculate zonal statistics for a multiband ImageCollection that requires no preprocessing, such as cloud masking or computation. Note that there is no built-in function for performing region reductions on ImageCollection objects. The zonalStats function that we are using for reduction is mapping the reduceRegions function over an ImageCollection.

    +

    A time series of MODIS eight-day surface reflectance composites demonstrates how to calculate zonal statistics for a multiband ImageCollection that requires no preprocessing, such as cloud masking or computation. Note that there is no built-in function for performing region reductions on ImageCollection objects. The zonalStats function that we are using for reduction is mapping the reduceRegions function over an ImageCollection.

    ###Buffer the Points

    -

    In this example, suppose the point collection represents center points for field plots that are 100 m x 100 m, and apply a 50 m radius buffer to the points to match the size of the plot. Since we want zonal statistics for square plots, set the second argument of the bufferPoints function to true, so that the bounds of the buffered points are returned.

    -

    var ptsModis = pts.map(bufferPoints(50, true));

    +

    In this example, suppose the point collection represents center points for field plots that are 100 m x 100 m, and apply a 50 m radius buffer to the points to match the size of the plot. Since we want zonal statistics for square plots, set the second argument of the bufferPoints function to true, so that the bounds of the buffered points are returned.

    +

    var ptsModis = pts.map(bufferPoints(50, true));

    ###Calculate Zonal Statistic

    Import the MODIS 500 m global eight-day surface reflectance composite collection and filter the collection to include data for July, August, and September from 2015 through 2019.

    -

    var modisCol = ee.ImageCollection(‘MODIS/006/MOD09A1’)
    -   .filterDate(‘2015-01-01’, ‘2020-01-01’)
    -   .filter(ee.Filter.calendarRange(183, 245, ‘DAY_OF_YEAR’));

    +

    var modisCol = ee.ImageCollection(‘MODIS/006/MOD09A1’)
    +.filterDate(‘2015-01-01’, ‘2020-01-01’)
    +.filter(ee.Filter.calendarRange(183, 245, ‘DAY_OF_YEAR’));

    Reduce each image in the collection by each plot according to the following parameters. Note that this time the reducer is defined as the neighborhood median (ee.Reducer.median) instead of the default mean, and that scale, CRS, and properties for the datetime are explicitly defined.

    -

    // Define parameters for the zonalStats function.
    -var params = {
    -   reducer: ee.Reducer.median(),
    -   scale: 500,
    -   crs: ‘EPSG:5070’,
    -   bands: [‘sur_refl_b01’, ‘sur_refl_b02’, ‘sur_refl_b06’],
    -   bandsRename: [‘modis_red’, ‘modis_nir’, ‘modis_swir’],
    -   datetimeName: ‘date’,
    -   datetimeFormat: ‘YYYY-MM-dd’
    -};

    -

    // Extract zonal statistics per point per image.
    -var ptsModisStats = zonalStats(modisCol, ptsModis, params);print(‘Limited MODIS zonal stats table’, ptsModisStats.limit(50));

    +
    // Define parameters for the zonalStats function.  
+var params = {  
+   reducer: ee.Reducer.median(),  
+   scale: 500,  
+   crs: 'EPSG:5070',  
+   bands: ['sur_refl_b01', 'sur_refl_b02', 'sur_refl_b06'],  
+   bandsRename: ['modis_red', 'modis_nir', 'modis_swir'],  
+   datetimeName: 'date',  
+   datetimeFormat: 'YYYY-MM-dd'  
+};  
+  
+// Extract zonal statistics per point per image.  
+var ptsModisStats = zonalStats(modisCol, ptsModis, params);print('Limited MODIS zonal stats table', ptsModisStats.limit(50));

    The result is a feature collection with a feature for all combinations of plots and images. Interpreted as a table, the result has 200 rows (5 plots times 40 images) and as many columns as there are feature properties. Feature properties include those from the plot asset and the image, and any associated non-system image properties. Note that the printed results are limited to the first 50 features for brevity.

    9.4.3 Landsat Time Series

    This example combines Landsat surface reflectance imagery across three instruments: Thematic Mapper (TM) from Landsat 5, Enhanced Thematic Mapper Plus (ETM+) from Landsat 7, and Operational Land Imager (OLI) from Landsat 8.

    -

    The following section prepares these collections so that band names are consistent and cloud masks are applied. Reflectance among corresponding bands are roughly congruent for the three sensors when using the surface reflectance product; therefore the processing steps that follow do not address inter-sensor harmonization. Review the current literature on inter-sensor harmonization practices if you’d like to apply a correction.

    +

    The following section prepares these collections so that band names are consistent and cloud masks are applied. Reflectance among corresponding bands are roughly congruent for the three sensors when using the surface reflectance product; therefore the processing steps that follow do not address inter-sensor harmonization. Review the current literature on inter-sensor harmonization practices if you’d like to apply a correction.

    ###Prepare the Landsat Image Collection

    -

    First, define the function to mask cloud and shadow pixels (See Chap. F4.3 for more detail on cloud masking).

    -

    // Mask clouds from images and apply scaling factors.
    -function maskScale(img) {   var qaMask = img.select(‘QA_PIXEL’).bitwiseAnd(parseInt(‘11111’,       2)).eq(0);   var saturationMask = img.select(‘QA_RADSAT’).eq(0);   // Apply the scaling factors to the appropriate bands.   var getFactorImg = function(factorNames) {       var factorList = img.toDictionary().select(factorNames)
    -           .values();       return ee.Image.constant(factorList);
    -   };   var scaleImg = getFactorImg([‘REFLECTANCE_MULT_BAND_.’]);   var offsetImg = getFactorImg([‘REFLECTANCE_ADD_BAND_.’]);   var scaled = img.select(‘SR_B.’).multiply(scaleImg).add(
    -   offsetImg);   // Replace the original bands with the scaled ones and apply the masks.   return img.addBands(scaled, null, true)
    -       .updateMask(qaMask)
    -       .updateMask(saturationMask);
    -}

    -

    Next, define functions to select and rename the bands of interest for the Operational Land Imager (OLI) aboard Landsat 8, and for the TM/ETM+ imagers aboard earlier Landsats. This is important because the band numbers are different for OLI and TM/ETM+, and it will make future index calculations easier.

    -

    // Selects and renames bands of interest for Landsat OLI.
    -function renameOli(img) {   return img.select(
    -       [‘SR_B2’, ‘SR_B3’, ‘SR_B4’, ‘SR_B5’, ‘SR_B6’, ‘SR_B7’],
    -       [‘Blue’, ‘Green’, ‘Red’, ‘NIR’, ‘SWIR1’, ‘SWIR2’]);
    -}

    -

    // Selects and renames bands of interest for TM/ETM+.
    -function renameEtm(img) {   return img.select(
    -       [‘SR_B1’, ‘SR_B2’, ‘SR_B3’, ‘SR_B4’, ‘SR_B5’, ‘SR_B7’],
    -       [‘Blue’, ‘Green’, ‘Red’, ‘NIR’, ‘SWIR1’, ‘SWIR2’]);
    -}

    -

    Combine the cloud mask and band renaming functions into preparation functions for OLI and TM/ETM+. Add any other sensor-specific preprocessing steps that you’d like to the functions below.

    -

    // Prepares (cloud masks and renames) OLI images.
    -function prepOli(img) {
    -   img = maskScale(img);
    -   img = renameOli(img);   return img;
    -}// Prepares (cloud masks and renames) TM/ETM+ images.
    -function prepEtm(img) {
    -   img = maskScale(img);
    -   img = renameEtm(img);   return img;
    -}

    -

    Get the Landsat surface reflectance collections for OLI, ETM+, and TM sensors. Filter them by the bounds of the point feature collection and apply the relevant image preparation function.

    -

    var ptsLandsat = pts.map(bufferPoints(15, true));

    -

    var oliCol = ee.ImageCollection(‘LANDSAT/LC08/C02/T1_L2’)
    -   .filterBounds(ptsLandsat)
    -   .map(prepOli);

    -

    var etmCol = ee.ImageCollection(‘LANDSAT/LE07/C02/T1_L2’)
    -   .filterBounds(ptsLandsat)
    -   .map(prepEtm);

    -

    var tmCol = ee.ImageCollection(‘LANDSAT/LT05/C02/T1_L2’)
    -   .filterBounds(ptsLandsat)
    -   .map(prepEtm);

    +

    First, define the function to mask cloud and shadow pixels (See Chap. F4.3 for more detail on cloud masking).

    +
    // Mask clouds from images and apply scaling factors.  
+function maskScale(img) {   var qaMask = img.select('QA_PIXEL').bitwiseAnd(parseInt('11111',       2)).eq(0);   var saturationMask = img.select('QA_RADSAT').eq(0);   // Apply the scaling factors to the appropriate bands.   var getFactorImg = function(factorNames) {       var factorList = img.toDictionary().select(factorNames)  
+           .values();       return ee.Image.constant(factorList);  
+   };   var scaleImg = getFactorImg(['REFLECTANCE_MULT_BAND_.']);   var offsetImg = getFactorImg(['REFLECTANCE_ADD_BAND_.']);   var scaled = img.select('SR_B.').multiply(scaleImg).add(  
+   offsetImg);   // Replace the original bands with the scaled ones and apply the masks.   return img.addBands(scaled, null, true)  
+       .updateMask(qaMask)  
+       .updateMask(saturationMask);  
+}
+
+Next, define functions to select and rename the bands of interest for the Operational Land Imager (OLI) aboard Landsat 8, and for the TM/ETM+ imagers aboard earlier Landsats. This is important because the band numbers are different for OLI and TM/ETM+, and it will make future index calculations easier.
+
+// Selects and renames bands of interest for Landsat OLI.  
+function renameOli(img) {   return img.select(  
+       ['SR_B2', 'SR_B3', 'SR_B4', 'SR_B5', 'SR_B6', 'SR_B7'],  
+       ['Blue', 'Green', 'Red', 'NIR', 'SWIR1', 'SWIR2']);  
+}  
+  
+// Selects and renames bands of interest for TM/ETM+.  
+function renameEtm(img) {   return img.select(  
+       ['SR_B1', 'SR_B2', 'SR_B3', 'SR_B4', 'SR_B5', 'SR_B7'],  
+       ['Blue', 'Green', 'Red', 'NIR', 'SWIR1', 'SWIR2']);  
+}
+
+Combine the cloud mask and band renaming functions into preparation functions for OLI and TM/ETM+. Add any other sensor-specific preprocessing steps that you’d like to the functions below.
+
+// Prepares (cloud masks and renames) OLI images.  
+function prepOli(img) {  
+   img = maskScale(img);  
+   img = renameOli(img);   return img;  
+}// Prepares (cloud masks and renames) TM/ETM+ images.  
+function prepEtm(img) {  
+   img = maskScale(img);  
+   img = renameEtm(img);   return img;  
+}
+
+Get the Landsat surface reflectance collections for OLI, ETM+, and TM sensors. Filter them by the bounds of the point feature collection and apply the relevant image preparation function.
+
+var ptsLandsat = pts.map(bufferPoints(15, true));  
+  
+var oliCol = ee.ImageCollection('LANDSAT/LC08/C02/T1_L2')  
+   .filterBounds(ptsLandsat)  
+   .map(prepOli);  
+  
+var etmCol = ee.ImageCollection('LANDSAT/LE07/C02/T1_L2')  
+   .filterBounds(ptsLandsat)  
+   .map(prepEtm);  
+  
+var tmCol = ee.ImageCollection('LANDSAT/LT05/C02/T1_L2')  
+   .filterBounds(ptsLandsat)  
+   .map(prepEtm);

    Merge the prepared sensor collections.

    -

    var landsatCol = oliCol.merge(etmCol).merge(tmCol);

    +

    var landsatCol = oliCol.merge(etmCol).merge(tmCol);

    ###Calculate Zonal Statistics

    -

    Reduce each image in the collection by each plot according to the following parameters. Note that this example defines the imgProps and imgPropsRename parameters to copy over and rename just two selected image properties: Landsat image ID and the satellite that collected the data. It also uses the max reducer, which, as an unweighted reducer, will return the maximum value from pixels that have their centroid within the buffer (see Sect. 4.1 below for more details).

    -

    // Define parameters for the zonalStats function.
    -var params = {
    -   reducer: ee.Reducer.max(),
    -   scale: 30,
    -   crs: ‘EPSG:5070’,
    -   bands: [‘Blue’, ‘Green’, ‘Red’, ‘NIR’, ‘SWIR1’, ‘SWIR2’],
    -   bandsRename: [‘ls_blue’, ‘ls_green’, ‘ls_red’, ‘ls_nir’,       ‘ls_swir1’, ‘ls_swir2’   ],
    -   imgProps: [‘SENSOR_ID’, ‘SPACECRAFT_ID’],
    -   imgPropsRename: [‘img_id’, ‘satellite’],
    -   datetimeName: ‘date’,
    -   datetimeFormat: ‘YYYY-MM-dd’
    -};

    -

    // Extract zonal statistics per point per image.
    -var ptsLandsatStats = zonalStats(landsatCol, ptsLandsat, params)   // Filter out observations where image pixels were all masked.   .filter(ee.Filter.notNull(params.bandsRename));
    -print(‘Limited Landsat zonal stats table’, ptsLandsatStats.limit(50));

    +

    Reduce each image in the collection by each plot according to the following parameters. Note that this example defines the imgProps and imgPropsRename parameters to copy over and rename just two selected image properties: Landsat image ID and the satellite that collected the data. It also uses the max reducer, which, as an unweighted reducer, will return the maximum value from pixels that have their centroid within the buffer (see Sect. 4.1 below for more details).

    +
    // Define parameters for the zonalStats function.  
+var params = {  
+   reducer: ee.Reducer.max(),  
+   scale: 30,  
+   crs: 'EPSG:5070',  
+   bands: ['Blue', 'Green', 'Red', 'NIR', 'SWIR1', 'SWIR2'],  
+   bandsRename: ['ls_blue', 'ls_green', 'ls_red', 'ls_nir',       'ls_swir1', 'ls_swir2'   ],  
+   imgProps: ['SENSOR_ID', 'SPACECRAFT_ID'],  
+   imgPropsRename: ['img_id', 'satellite'],  
+   datetimeName: 'date',  
+   datetimeFormat: 'YYYY-MM-dd'  
+};  
+  
+// Extract zonal statistics per point per image.  
+var ptsLandsatStats = zonalStats(landsatCol, ptsLandsat, params)   // Filter out observations where image pixels were all masked.   .filter(ee.Filter.notNull(params.bandsRename));  
+print('Limited Landsat zonal stats table', ptsLandsatStats.limit(50));

    The result is a feature collection with a feature for all combinations of plots and images.

    ###Dealing with Large Collections

    -

    If your browser times out, try exporting the results (as described in Chap. F6.2). It’s likely that point feature collections that cover a large area or contain many points (point-image observations) will need to be exported as a batch task by either exporting the final feature collection as an asset or as a CSV/shapefile/GeoJSON to Google Drive or GCS.

    -

    Here is how you would export the above Landsat image-point feature collection to an asset and to Google Drive. Run the following code, activate the Code Editor Tasks tab, and then click the Run button. If you don’t specify your own existing folder in Drive, the folder “EEFA_outputs” will be created.

    +

    If your browser times out, try exporting the results (as described in Chap. F6.2). It’s likely that point feature collections that cover a large area or contain many points (point-image observations) will need to be exported as a batch task by either exporting the final feature collection as an asset or as a CSV/shapefile/GeoJSON to Google Drive or GCS.

    +

    Here is how you would export the above Landsat image-point feature collection to an asset and to Google Drive. Run the following code, activate the Code Editor Tasks tab, and then click the Run button. If you don’t specify your own existing folder in Drive, the folder “EEFA_outputs” will be created.

    Export.table.toAsset({
    -   collection: ptsLandsatStats,
    -   description: ‘EEFA_export_Landsat_to_points’,
    -   assetId: ‘EEFA_export_values_to_points’
    +collection: ptsLandsatStats,
    +description: ‘EEFA_export_Landsat_to_points’,
    +assetId: ‘EEFA_export_values_to_points’
    });

    Export.table.toDrive({
    -   collection: ptsLandsatStats,
    -   folder: ‘EEFA_outputs’, // this will create a new folder if it doesn’t exist   description: ‘EEFA_export_values_to_points’,
    -   fileFormat: ‘CSV’
    +collection: ptsLandsatStats,
    +folder: ‘EEFA_outputs’, // this will create a new folder if it doesn’t exist description: ‘EEFA_export_values_to_points’,
    +fileFormat: ‘CSV’
    });

    @@ -1526,7 +1724,7 @@ Note
    -

    Code Checkpoint F52b. The book’s repository contains a script that shows what your code should look like at this point.

    +

    Code Checkpoint F52b. The book’s repository contains a script that shows what your code should look like at this point.

    @@ -1535,87 +1733,99 @@ Note

    9.5 Additional Notes

    9.5.1 Weighted Versus Unweighted Region Reduction

    -

    A region used for calculation of zonal statistics often bisects multiple pixels. Should partial pixels be included in zonal statistics? Earth Engine lets you decide by allowing you to define a reducer as either weighted or unweighted (or you can provide per-pixel weight specification as an image band). A weighted reducer will include partial pixels in the zonal statistic calculation by weighting each pixel’s contribution according to the fraction of the area intersecting the region. An unweighted reducer, on the other hand, gives equal weight to all pixels whose cell center intersects the region; all other pixels are excluded from calculation of the statistic.

    -

    For aggregate reducers like ee.Reducer.mean and ee.Reducer.median, the default mode is weighted, while identifier reducers such as ee.Reducer.min and ee.Reducer.max are unweighted. You can adjust the behavior of weighted reducers by calling unweighted on them, as in ee.Reducer.mean.unweighted. You may also specify the weights by modifying the reducer with splitWeights; however, that is beyond the scope of this book.

    +

    A region used for calculation of zonal statistics often bisects multiple pixels. Should partial pixels be included in zonal statistics? Earth Engine lets you decide by allowing you to define a reducer as either weighted or unweighted (or you can provide per-pixel weight specification as an image band). A weighted reducer will include partial pixels in the zonal statistic calculation by weighting each pixel’s contribution according to the fraction of the area intersecting the region. An unweighted reducer, on the other hand, gives equal weight to all pixels whose cell center intersects the region; all other pixels are excluded from calculation of the statistic.

    +

    For aggregate reducers like ee.Reducer.mean and ee.Reducer.median, the default mode is weighted, while identifier reducers such as ee.Reducer.min and ee.Reducer.max are unweighted. You can adjust the behavior of weighted reducers by calling unweighted on them, as in ee.Reducer.mean.unweighted. You may also specify the weights by modifying the reducer with splitWeights; however, that is beyond the scope of this book.

    9.5.2 Copy Properties to Computed Images

    Derived, computed images do not retain the properties of their source image, so be sure to copy properties to computed images if you want them included in the region reduction table. For instance, consider the simple computation of unscaling Landsat SR data:

    -

    // Define a Landsat image.
    -var img = ee.ImageCollection(‘LANDSAT/LC08/C02/T1_L2’).first();

    -

    // Print its properties.
    -print(‘All image properties’, img.propertyNames());

    -

    // Subset the reflectance bands and unscale them.
    -var computedImg = img.select(‘SR_B.’).multiply(0.0000275).add(-0.2);

    -

    // Print the unscaled image’s properties.
    -print(‘Lost original image properties’, computedImg.propertyNames());

    -

    Notice how the computed image does not have the source image’s properties and only retains the bands information. To fix this, use the copyProperties function to add desired source properties to the derived image. It is best practice to copy only the properties you really need because some properties, such as those containing geometry objects, lists, or feature collections, can significantly increase the computational burden for large collections.

    -

    // Subset the reflectance bands and unscale them, keeping selected
    -// source properties.
    -var computedImg = img.select(‘SR_B.’).multiply(0.0000275).add(-0.2)
    -   .copyProperties(img, [‘system:time_start’, ‘LANDSAT_PRODUCT_ID’]);

    -

    // Print the unscaled image’s properties.
    -print(‘Selected image properties retained’, computedImg
    -.propertyNames());

    +
    // Define a Landsat image.  
+var img = ee.ImageCollection('LANDSAT/LC08/C02/T1_L2').first();  
+  
+// Print its properties.  
+print('All image properties', img.propertyNames());  
+  
+// Subset the reflectance bands and unscale them.  
+var computedImg = img.select('SR_B.').multiply(0.0000275).add(-0.2);  
+  
+// Print the unscaled image's properties.  
+print('Lost original image properties', computedImg.propertyNames());
    +

    Notice how the computed image does not have the source image’s properties and only retains the bands information. To fix this, use the copyProperties function to add desired source properties to the derived image. It is best practice to copy only the properties you really need because some properties, such as those containing geometry objects, lists, or feature collections, can significantly increase the computational burden for large collections.

    +
    // Subset the reflectance bands and unscale them, keeping selected  
+// source properties.  
+var computedImg = img.select('SR_B.').multiply(0.0000275).add(-0.2)  
+   .copyProperties(img, ['system:time_start', 'LANDSAT_PRODUCT_ID']);  
+  
+// Print the unscaled image's properties.  
+print('Selected image properties retained', computedImg  
+.propertyNames());

    Now selected properties are included. Use this technique when returning computed, derived images in a mapped function, and in single-image operations.

    9.5.3 Understanding Which Pixels are Included in Polygon Statistics

    If you want to visualize what pixels are included in a polygon for a region reducer, you can adapt the following code to use your own region (by replacing geometry), dataset, desired scale, and CRS parameters. The important part to note is that the image data you are adding to the map is reprojected using the same scale and CRS as that used in your region reduction (see Fig. F5.2.3).

    -

    // Define polygon geometry.
    -var geometry = ee.Geometry.Polygon(
    -   [
    -       [
    -           [-118.6019835717645, 37.079867782687884],
    -           [-118.6019835717645, 37.07838698844939],
    -           [-118.60036351751951, 37.07838698844939],
    -           [-118.60036351751951, 37.079867782687884]
    -       ]
    -   ], null, false);

    -

    // Import the MERIT global elevation dataset.
    -var elev = ee.Image(‘MERIT/DEM/v1_0_3’);

    -

    // Define desired scale and crs for region reduction (for image display too).
    -var proj = {
    -   scale: 90,
    -   crs: ‘EPSG:5070’
    -};

    -

    The count reducer will return how many pixel centers are overlapped by the polygon region, which would be the number of pixels included in any unweighted reducer statistic. You can also visualize which pixels will be included in the reduction by using the toCollection reducer on a latitude/longitude image and adding resulting coordinates as feature geometry. Be sure to specify CRS and scale for both the region reducers and the reprojected layer added to the map (see bullet list below for more details).

    -

    // A count reducer will return how many pixel centers are overlapped by the
    -// polygon region.
    -var count = elev.select(0).reduceRegion({
    -   reducer: ee.Reducer.count(),
    -   geometry: geometry,
    -   scale: proj.scale,   crs: proj.crs
    -});
    -print(‘n pixels in the reduction’, count.get(‘dem’));

    -

    // Make a feature collection of pixel center points for those that are
    -// included in the reduction.
    -var pixels = ee.Image.pixelLonLat().reduceRegion({
    -   reducer: ee.Reducer.toCollection([‘lon’, ‘lat’]),
    -   geometry: geometry,
    -    scale: proj.scale,   crs: proj.crs
    -});
    -var pixelsFc = ee.FeatureCollection(pixels.get(‘features’)).map(   function(f) {       return f.setGeometry(ee.Geometry.Point([f.get(‘lon’), f
    -           .get(‘lat’)
    -       ]));
    -   });

    -

    // Display layers on the map.
    -Map.centerObject(geometry, 18);
    -Map.addLayer(
    -   elev.reproject({
    -       crs: proj.crs,
    -       scale: proj.scale   }),
    -   {
    -       min: 2500,
    -       max: 3000,
    -       palette: [‘blue’, ‘white’, ‘red’]
    -   }, ‘Image’);
    -Map.addLayer(geometry, {
    -   color: ‘white’}, ‘Geometry’);
    -Map.addLayer(pixelsFc, {
    -   color: ‘purple’}, ‘Pixels in reduction’);

    -

    -

    Fig. F5.2.3 Identifying pixels used in zonal statistics. By mapping the image and vector together, you can see which pixels are included in the unweighted statistic. For this example, three pixels would be included in the statistic because the polygon covers the center point of three pixels.

    +
    // Define polygon geometry.  
+var geometry = ee.Geometry.Polygon(  
+   [  
+       [  
+           [-118.6019835717645, 37.079867782687884],  
+           [-118.6019835717645, 37.07838698844939],  
+           [-118.60036351751951, 37.07838698844939],  
+           [-118.60036351751951, 37.079867782687884]  
+       ]  
+   ], null, false);  
+  
+// Import the MERIT global elevation dataset.  
+var elev = ee.Image('MERIT/DEM/v1_0_3');  
+  
+// Define desired scale and crs for region reduction (for image display too).  
+var proj = {  
+   scale: 90,  
+   crs: 'EPSG:5070'  
+};
    +

    The count reducer will return how many pixel centers are overlapped by the polygon region, which would be the number of pixels included in any unweighted reducer statistic. You can also visualize which pixels will be included in the reduction by using the toCollection reducer on a latitude/longitude image and adding resulting coordinates as feature geometry. Be sure to specify CRS and scale for both the region reducers and the reprojected layer added to the map (see bullet list below for more details).

    +
    // A count reducer will return how many pixel centers are overlapped by the  
+// polygon region.  
+var count = elev.select(0).reduceRegion({  
+   reducer: ee.Reducer.count(),  
+   geometry: geometry,  
+   scale: proj.scale,   crs: proj.crs  
+});  
+print('n pixels in the reduction', count.get('dem'));  
+  
+// Make a feature collection of pixel center points for those that are  
+// included in the reduction.  
+var pixels = ee.Image.pixelLonLat().reduceRegion({  
+   reducer: ee.Reducer.toCollection(['lon', 'lat']),  
+   geometry: geometry,  
+    scale: proj.scale,   crs: proj.crs  
+});  
+var pixelsFc = ee.FeatureCollection(pixels.get('features')).map(   function(f) {       return f.setGeometry(ee.Geometry.Point([f.get('lon'), f  
+           .get('lat')  
+       ]));  
+   });  
+  
+// Display layers on the map.  
+Map.centerObject(geometry, 18);  
+Map.addLayer(  
+   elev.reproject({  
+       crs: proj.crs,  
+       scale: proj.scale   }),  
+   {  
+       min: 2500,  
+       max: 3000,  
+       palette: ['blue', 'white', 'red']  
+   }, 'Image');  
+Map.addLayer(geometry, {  
+   color: 'white'}, 'Geometry');  
+Map.addLayer(pixelsFc, {  
+   color: 'purple'}, 'Pixels in reduction');
    +
    +
    +

    +

    Fig. F5.2.3 Identifying pixels used in zonal statistics. By mapping the image and vector together, you can see which pixels are included in the unweighted statistic. For this example, three pixels would be included in the statistic because the polygon covers the center point of three pixels.

    +
    +
    @@ -1626,32 +1836,32 @@ Note
    -

    Code Checkpoint F52c. The book’s repository contains a script that shows what your code should look like at this point.

    +

    Code Checkpoint F52c. The book’s repository contains a script that shows what your code should look like at this point.

    Finally, here are some notes on CRS and scale:

      -
    • Earth Engine runs reduceRegion using the projection of the image’s first band if the CRS is unspecified in the function. For imagery spanning multiple UTM zones, for example, this would lead to different origins. For some functions Earth Engine uses the default EPSG:4326. Therefore, when the opportunity is presented, such as by the reduceRegion function, it is important to specify the scale and CRS explicitly.
      • -
    • The Map default CRS is EPSG:3857. When looking closely at pixels on the map, the data layer scale and CRS should also be set explicitly. Note that zooming out after setting a relatively small scale when reprojecting may result in memory and/or timeout errors because optimized pyramid layers for each zoom level will not be used.
      • -
    • Specifying the CRS and scale in both the reduceRegion and addLayer functions allows the map visualization to align with the information printed in the Console.
      • +
    • Earth Engine runs reduceRegion using the projection of the image’s first band if the CRS is unspecified in the function. For imagery spanning multiple UTM zones, for example, this would lead to different origins. For some functions Earth Engine uses the default EPSG:4326. Therefore, when the opportunity is presented, such as by the reduceRegion function, it is important to specify the scale and CRS explicitly.
      • +
    • The Map default CRS is EPSG:3857. When looking closely at pixels on the map, the data layer scale and CRS should also be set explicitly. Note that zooming out after setting a relatively small scale when reprojecting may result in memory and/or timeout errors because optimized pyramid layers for each zoom level will not be used.
      • +
    • Specifying the CRS and scale in both the reduceRegion and addLayer functions allows the map visualization to align with the information printed in the Console.
    • The Earth Engine default, WGS 84 lat long (EPSG:4326), is a generic CRS that works worldwide. The code above reprojects to EPSG:5070, North American Equal Albers, which is a CRS that preserves area for North American locations. Use the CRS that is best for your use case when adapting this to your own project, or maintain (and specify) the CRS of the image using, for example, crs: ‘img.projection().crs()’.

    Synthesis

    -

    Question 1. Look at the MODIS example (Sect. 3.2), which uses the median reducer. Try modifying the reducer to be unweighted, either by specifying unweighted or using an identifier reducer like max. What happens, and why?

    +

    Question 1. Look at the MODIS example (Sect. 3.2), which uses the median reducer. Try modifying the reducer to be unweighted, either by specifying unweighted or using an identifier reducer like max. What happens, and why?

    Question 2. Calculate zonal statistics for your own buffered points or polygons using a raster and reducer of interest. Be sure to consider the spatial scale of the raster and whether a weighted or unweighted reducer would be more appropriate for your interests.

    If the point or polygon file is stored in a local shapefile or CSV file, first upload the data to your Earth Engine assets. All columns in your vector file, such as the plot name, will be retained through this process. Once you have an Earth Engine table asset ready, import the asset into your script by hovering over the name of the asset and clicking the arrow at the right side, or by calling it in your script with the following code.

    -

    var pts = ee.FeatureCollection(‘users/yourUsername/yourAsset’);

    +

    var pts = ee.FeatureCollection(‘users/yourUsername/yourAsset’);

    If you prefer to define points or polygons dynamically rather than loading an asset, you can add them to your script using the geometry tools. See Chap. F2.1 and F5.0 for more detail on adding and creating vector data.

    -

    Question 3. Try the code from Sect. 4.3 using the MODIS data and the first point from the pts variable. Among other modifications, you will need to create a buffer for the point, take a single MODIS image from the collection, and change visualization parameters.

    +

    Question 3. Try the code from Sect. 4.3 using the MODIS data and the first point from the pts variable. Among other modifications, you will need to create a buffer for the point, take a single MODIS image from the collection, and change visualization parameters.

    • Think about the CRS in the code: The code reprojects to EPSG:5070, but MODIS is collected in the sinusoidal projection SR-ORG:6974. Try that CRS and describe how the image changes.
    • Is the count reducer weighted or unweighted? Give an example of a circumstance to use a weighted reducer and an example for an unweighted reducer. Specify the buffer size you would use and the spatial resolution of your dataset.
    -

    Question 4. In the examples above, only a single ee.Reducer is passed to the zonalStats function, which means that only a single statistic is calculated (for example, zonal mean or median or maximum). What if you want multiple statistics—can you alter the code in Sect. 3.1 to (1) make the point buffer 500 instead of 45; (2) add the reducer parameter to the params dictionary; and (3) as its argument, supply a combined ee.Reducer that will calculate minimum, maximum, standard deviation, and mean statistics?

    -

    To achieve this you’ll need to chain several ee.Reducer.combine functions together. Note that if you accept all the individual ee.Reducer and ee.Reducer.combine function defaults, you’ll run into two problems related to reducer weighting differences, and whether or not the image inputs are shared among the combined set of reducers. How can you manipulate the individual ee.Reducer and ee.Reducer.combine functions to achieve the goal of calculating multiple zonal statistics in one call to the zonalStats function?

    +

    Question 4. In the examples above, only a single ee.Reducer is passed to the zonalStats function, which means that only a single statistic is calculated (for example, zonal mean or median or maximum). What if you want multiple statistics—can you alter the code in Sect. 3.1 to (1) make the point buffer 500 instead of 45; (2) add the reducer parameter to the params dictionary; and (3) as its argument, supply a combined ee.Reducer that will calculate minimum, maximum, standard deviation, and mean statistics?

    +

    To achieve this you’ll need to chain several ee.Reducer.combine functions together. Note that if you accept all the individual ee.Reducer and ee.Reducer.combine function defaults, you’ll run into two problems related to reducer weighting differences, and whether or not the image inputs are shared among the combined set of reducers. How can you manipulate the individual ee.Reducer and ee.Reducer.combine functions to achieve the goal of calculating multiple zonal statistics in one call to the zonalStats function?

    Conclusion

    @@ -1665,15 +1875,13 @@ Note

    10 Advanced Vector Operations

    -

    ::: {.callout-tip} # Chapter Information

    +

    :::{.callout-tip} # Chapter Information

    Author

    -

     

    Ujaval Gandhi

    Overview

    -

     

    This chapter covers advanced techniques for visualizing and analyzing vector data in Earth Engine. There are many ways to visualize feature collections, and you will learn how to pick the appropriate method to create visualizations, such as a choropleth map. We will also cover geoprocessing techniques involving multiple vector layers, such as selecting features in one layer by their proximity to features in another layer and performing spatial joins.

    @@ -1687,80 +1895,93 @@ Note

    Assumes you know how to:

      -
    • Filter a FeatureCollection to obtain a subset (Chap. F5.0, Chap. F5.1).
      • -
    • Write a function and map it over a FeatureCollection (Chap. F5.1, Chap. F5.2).
      • +
    • Filter a FeatureCollection to obtain a subset (Chap. F5.0, Chap. F5.1).
      • +
    • Write a function and map it over a FeatureCollection (Chap. F5.1, Chap. F5.2).

    10.1 Visualizing Feature Collections

    -

    There is a distinct difference between how rasters and vectors are visualized. While images are typically visualized based on pixel values, vector layers use feature properties (i.e., attributes) to create a visualization. Vector layers are rendered on the Map by assigning a value to the red, green, and blue channels for each pixel on the screen based on the geometry and attributes of the features. The functions used for vector data visualization in Earth Engine are listed below in increasing order of complexity.

    +

    There is a distinct difference between how rasters and vectors are visualized. While images are typically visualized based on pixel values, vector layers use feature properties (i.e., attributes) to create a visualization. Vector layers are rendered on the Map by assigning a value to the red, green, and blue channels for each pixel on the screen based on the geometry and attributes of the features. The functions used for vector data visualization in Earth Engine are listed below in increasing order of complexity.

      -
    • Map.addLayer: As with raster layers, you can add a FeatureCollection to the Map by specifying visualization parameters. This method supports only one visualization parameter: color. All features are rendered with the specified color.
      • -
    • draw: This function supports the parameters pointRadius and strokeWidth in addition to color. It renders all features of the layer with the specified parameters.
      • -
    • paint: This is a more powerful function that can render each feature with a different color and width based on the values in the specified property.
      • +
    • Map.addLayer: As with raster layers, you can add a FeatureCollection to the Map by specifying visualization parameters. This method supports only one visualization parameter: color. All features are rendered with the specified color.
      • +
    • draw: This function supports the parameters pointRadius and strokeWidth in addition to color. It renders all features of the layer with the specified parameters.
      • +
    • paint: This is a more powerful function that can render each feature with a different color and width based on the values in the specified property.
    • style: This is the most versatile function. It can apply a different style to each feature, including color, pointSize, pointShape, width, fillColor, and lineType.

    In the exercises below, we will learn how to use each of these functions and see how they can generate different types of maps.

    10.1.1 Creating a Choropleth Map

    -

    We will use the TIGER: US Census Blocks layer, which stores census block boundaries and their characteristics within the United States, along with the San Francisco neighborhoods layer from Chap. F5.0 to create a population density map for the city of San Francisco.

    -

    We start by loading the census blocks and San Francisco neighborhoods layers. We use ee.Filter.bounds to filter the census blocks layer to the San Francisco boundary.

    -

    var blocks = ee.FeatureCollection(‘TIGER/2010/Blocks’);
    -var roads = ee.FeatureCollection(‘TIGER/2016/Roads’);
    -var sfNeighborhoods = ee.FeatureCollection(   ‘projects/gee-book/assets/F5-0/SFneighborhoods’);

    -

    var geometry = sfNeighborhoods.geometry();
    +

    We will use the TIGER: US Census Blocks layer, which stores census block boundaries and their characteristics within the United States, along with the San Francisco neighborhoods layer from Chap. F5.0 to create a population density map for the city of San Francisco.

    +

    We start by loading the census blocks and San Francisco neighborhoods layers. We use ee.Filter.bounds to filter the census blocks layer to the San Francisco boundary.

    +

    var blocks = ee.FeatureCollection(‘TIGER/2010/Blocks’);
    +var roads = ee.FeatureCollection(‘TIGER/2016/Roads’);
    +var sfNeighborhoods = ee.FeatureCollection( ‘projects/gee-book/assets/F5-0/SFneighborhoods’);

    +

    var geometry = sfNeighborhoods.geometry();
    Map.centerObject(geometry);

    -

    // Filter blocks to the San Francisco boundary.
    -var sfBlocks = blocks.filter(ee.Filter.bounds(geometry));

    -

    The simplest way to visualize this layer is to use Map.addLayer (Fig. F5.3.1). We can specify a color value in the visParams parameter of the function. Each census block polygon will be rendered with stroke and fill of the specified color. The fill color is the same as the stroke color but has a 66% opacity.

    -

    // Visualize with a single color.
    -Map.addLayer(sfBlocks, {
    -   color: ‘#de2d26’}, ‘Census Blocks (single color)’);

    -

    -

    Fig. F5.3.1 San Francisco census blocks

    -

    The census blocks table has a property named ‘pop10’ containing the population totals as of the 2010 census. We can use this to create a choropleth map showing population density. We first need to compute the population density for each feature and add it as a property. To add a new property to each feature, we can map a function over the FeatureCollection and calculate the new property called ‘pop_density’. Earth Engine provides the area function, which can calculate the area of a feature in square meters. We convert it to square miles and calculate the population density per square mile.

    -

    // Add a pop_density column.
    -var sfBlocks = sfBlocks.map(function(f) {   // Get the polygon area in square miles.   var area_sqmi = f.area().divide(2.59e6);   var population = f.get(‘pop10’);   // Calculate population density.   var density = ee.Number(population).divide(area_sqmi);   return f.set({       ‘area_sqmi’: area_sqmi,       ‘pop_density’: density
    -   });
    -});

    -

    Now we can use the paint function to create an image from this FeatureCollection using the pop_density property. The paint function needs an empty image that needs to be cast to the appropriate data type. Let’s use the aggregate_stats function to calculate basic statistics for the given column of a FeatureCollection.

    -

    // Calculate the statistics of the newly computed column.
    -var stats = sfBlocks.aggregate_stats(‘pop_density’);
    -print(stats);

    +
    // Filter blocks to the San Francisco boundary.  
+var sfBlocks = blocks.filter(ee.Filter.bounds(geometry));
    +

    The simplest way to visualize this layer is to use Map.addLayer (Fig. F5.3.1). We can specify a color value in the visParams parameter of the function. Each census block polygon will be rendered with stroke and fill of the specified color. The fill color is the same as the stroke color but has a 66% opacity.

    +
    // Visualize with a single color.  
+Map.addLayer(sfBlocks, {  
+   color: '#de2d26'}, 'Census Blocks (single color)');
    +
    +
    +

    +

    Fig. F5.3.1 San Francisco census blocks

    +
    +
    +

    The census blocks table has a property named ‘pop10’ containing the population totals as of the 2010 census. We can use this to create a choropleth map showing population density. We first need to compute the population density for each feature and add it as a property. To add a new property to each feature, we can map a function over the FeatureCollection and calculate the new property called ‘pop_density’. Earth Engine provides the area function, which can calculate the area of a feature in square meters. We convert it to square miles and calculate the population density per square mile.

    +
    // Add a pop_density column.  
+var sfBlocks = sfBlocks.map(function(f) {   // Get the polygon area in square miles.   var area_sqmi = f.area().divide(2.59e6);   var population = f.get('pop10');   // Calculate population density.   var density = ee.Number(population).divide(area_sqmi);   return f.set({       'area_sqmi': area_sqmi,       'pop_density': density  
+   });  
+});
    +

    Now we can use the paint function to create an image from this FeatureCollection using the pop_density property. The paint function needs an empty image that needs to be cast to the appropriate data type. Let’s use the aggregate_stats function to calculate basic statistics for the given column of a FeatureCollection.

    +
    // Calculate the statistics of the newly computed column.  
+var stats = sfBlocks.aggregate_stats('pop_density');  
+print(stats);

    You will see that the population density values have a large range. We also have values that are greater than 100,000, so we need to make sure we select a data type that can store values of this size. We create an empty image and cast it to int32, which is able to hold large integer values.

    D

    -

    The result is an image with pixel values representing the population density of the polygons. We can now use the standard image visualization method to add this layer to the Map (Fig. F5.3.2). Then, we need to determine minimum and maximum values for the visualization parameters.A reliable technique to produce a good visualization is to find minimum and maximum values that are within one standard deviation. From the statistics that we calculated earlier, we can estimate good minimum and maximum values to be 0 and 50000, respectively.

    -

    var palette = [‘fee5d9’, ‘fcae91’, ‘fb6a4a’, ‘de2d26’, ‘a50f15’];
    -var visParams = {
    -   min: 0,
    -   max: 50000,
    -   palette: palette
    +

    The result is an image with pixel values representing the population density of the polygons. We can now use the standard image visualization method to add this layer to the Map (Fig. F5.3.2). Then, we need to determine minimum and maximum values for the visualization parameters.A reliable technique to produce a good visualization is to find minimum and maximum values that are within one standard deviation. From the statistics that we calculated earlier, we can estimate good minimum and maximum values to be 0 and 50000, respectively.

    +

    var palette = [‘fee5d9’, ‘fcae91’, ‘fb6a4a’, ‘de2d26’, ‘a50f15’];
    +var visParams = {
    +min: 0,
    +max: 50000,
    +palette: palette
    };
    -Map.addLayer(sfBlocksPaint.clip(geometry), visParams,   ‘Population Density’);

    -

    -

    Fig. F5.3.2 San Francisco population density

    +Map.addLayer(sfBlocksPaint.clip(geometry), visParams, ‘Population Density’);

    +
    +
    +

    +

    Fig. F5.3.2 San Francisco population density

    +
    +

    10.1.2 Creating a Categorical Map

    -

    Continuing the exploration of styling methods, we will now learn about draw and style. These are the preferred methods of styling for points and line layers. Let’s see how we can visualize the TIGER: US Census Roads layer to create a categorical map.

    -

    We start by filtering the roads layer to the San Francisco boundary and using Map.addLayer to visualize it.

    -

    // Filter roads to San Francisco boundary.
    -var sfRoads = roads.filter(ee.Filter.bounds(geometry));

    -

    Map.addLayer(sfRoads, {
    -   color: ‘blue’}, ‘Roads (default)’);

    -

    The default visualization renders each line using a width of 2 pixels. The draw function provides a way to specify a different line width. Let’s use it to render the layer with the same color as before but with a line width of 1 pixel (Fig. F5.3.3).

    -

    // Visualize with draw().
    -var sfRoadsDraw = sfRoads.draw({
    -   color: ‘blue’,
    -   strokeWidth: 1
    -});
    -Map.addLayer(sfRoadsDraw, {}, ‘Roads (Draw)’);

    +

    Continuing the exploration of styling methods, we will now learn about draw and style. These are the preferred methods of styling for points and line layers. Let’s see how we can visualize the TIGER: US Census Roads layer to create a categorical map.

    +

    We start by filtering the roads layer to the San Francisco boundary and using Map.addLayer to visualize it.

    +
    // Filter roads to San Francisco boundary.  
+var sfRoads = roads.filter(ee.Filter.bounds(geometry));  
+  
+Map.addLayer(sfRoads, {  
+   color: 'blue'}, 'Roads (default)');
    +

    The default visualization renders each line using a width of 2 pixels. The draw function provides a way to specify a different line width. Let’s use it to render the layer with the same color as before but with a line width of 1 pixel (Fig. F5.3.3).

    +
    // Visualize with draw().  
+var sfRoadsDraw = sfRoads.draw({  
+   color: 'blue',  
+   strokeWidth: 1  
+});  
+Map.addLayer(sfRoadsDraw, {}, 'Roads (Draw)');

    -

    -

    Fig. F5.3.3 San Francisco roads rendered with a line width of 2 pixels (left) and and a line width of 1 pixel (right)

    -

    The road layer has a column called “MTFCC” (standing for the MAF/TIGER Feature Class Code). This contains the road priority codes, representing the various types of roads, such as primary and secondary. We can use this information to render each road segment according to its priority. The draw function doesn’t allow us to specify different styles for each feature. Instead, we need to make use of the style function.

    -

    The column contains string values indicating different road types as indicated in Table F5.3.1. This full list is available at the MAF/TIGER Feature Class Code Definitions page on the US Census Bureau website.

    -

    Table F5.3.1 Census Bureau road priority codes

    +
    +
    +

    +

    Fig. F5.3.3 San Francisco roads rendered with a line width of 2 pixels (left) and and a line width of 1 pixel (right)

    +
    +
    +

    The road layer has a column called “MTFCC” (standing for the MAF/TIGER Feature Class Code). This contains the road priority codes, representing the various types of roads, such as primary and secondary. We can use this information to render each road segment according to its priority. The draw function doesn’t allow us to specify different styles for each feature. Instead, we need to make use of the style function.

    +

    The column contains string values indicating different road types as indicated in Table F5.3.1. This full list is available at the MAF/TIGER Feature Class Code Definitions page on the US Census Bureau website.

    +

    Table F5.3.1 Census Bureau road priority codes

    MTFCC

    Feature Class

    S1100

    @@ -1794,7 +2015,7 @@ Map.addLayer(sfRoadsDraw, {}, ‘Roads (Draw)’);

    S2000

    Road Median

    Let’s say we want to create a map with rules based on the MTFCC values shown in Table F5.3.2.

    -

    Table F5.3.2 Styling Parameters for Road Priority Codes

    +

    Table F5.3.2 Styling Parameters for Road Priority Codes

    MTFCC

    Color

    Line Width

    @@ -1811,20 +2032,24 @@ Map.addLayer(sfRoadsDraw, {}, ‘Roads (Draw)’);

    Gray

    1

    Let’s define a dictionary containing the styling information.

    -

    var styles = ee.Dictionary({   ‘S1100’: {       ‘color’: ‘blue’,       ‘width’: 3   },   ‘S1200’: {       ‘color’: ‘green’,       ‘width’: 2   },   ‘S1400’: {       ‘color’: ‘orange’,       ‘width’: 1   }
    -});var defaultStyle = {
    -   color: ‘gray’,   ‘width’: 1
    +

    var styles = ee.Dictionary({ ‘S1100’: { ‘color’: ‘blue’, ‘width’: 3 }, ‘S1200’: { ‘color’: ‘green’, ‘width’: 2 }, ‘S1400’: { ‘color’: ‘orange’, ‘width’: 1 }
    +});var defaultStyle = {
    +color: ‘gray’, ‘width’: 1
    };

    -

    The style function needs a property in the FeatureCollection that contains a dictionary with the style parameters. This allows you to specify a different style for each feature. To create a new property, we map a function over the FeatureCollection and assign an appropriate style dictionary to a new property named ‘style’. Note the use of the get function, which allows us to fetch the value for a key in the dictionary. It also takes a default value in case the specified key does not exist. We make use of this to assign different styles to the three road classes specified in Table 5.3.2 and a default style to all others.

    -

    var sfRoads = sfRoads.map(function(f) {   var classcode = f.get(‘mtfcc’);   var style = styles.get(classcode, defaultStyle);   return f.set(‘style’, style);
    +

    The style function needs a property in the FeatureCollection that contains a dictionary with the style parameters. This allows you to specify a different style for each feature. To create a new property, we map a function over the FeatureCollection and assign an appropriate style dictionary to a new property named ‘style’. Note the use of the get function, which allows us to fetch the value for a key in the dictionary. It also takes a default value in case the specified key does not exist. We make use of this to assign different styles to the three road classes specified in Table 5.3.2 and a default style to all others.

    +

    var sfRoads = sfRoads.map(function(f) { var classcode = f.get(‘mtfcc’); var style = styles.get(classcode, defaultStyle); return f.set(‘style’, style);
    });

    -

    Our collection is now ready to be styled. We call the style function to specify the property that contains the dictionary of style parameters. The output of the style function is an RGB image rendered from the FeatureCollection (Fig. F5.3.4).

    -

    var sfRoadsStyle = sfRoads.style({
    -   styleProperty: ‘style’
    +

    Our collection is now ready to be styled. We call the style function to specify the property that contains the dictionary of style parameters. The output of the style function is an RGB image rendered from the FeatureCollection (Fig. F5.3.4).

    +

    var sfRoadsStyle = sfRoads.style({
    +styleProperty: ‘style’
    });
    Map.addLayer(sfRoadsStyle.clip(geometry), {}, ‘Roads (Style)’);

    -

    -

    Fig. F5.3.4 San Francisco roads rendered according to road priority

    +
    +
    +

    +

    Fig. F5.3.4 San Francisco roads rendered according to road priority

    +
    +
    @@ -1835,7 +2060,7 @@ Note
    -

    Code Checkpoint F53a. The book’s repository contains a script that shows what your code should look like at this point.

    +

    Code Checkpoint F53a. The book’s repository contains a script that shows what your code should look like at this point.

    Save your script for your own future use, as outlined in Chap. F1.0. Then, refresh the Code Editor to begin with a new script for the next section.

    @@ -1844,123 +2069,149 @@ Note

    10.2 Joins with Feature Collections

    Earth Engine was designed as a platform for processing raster data, and that is where it shines. Over the years, it has acquired advanced vector data processing capabilities, and users are now able to carry out complex geoprocessing tasks within Earth Engine. You can leverage the distributed processing power of Earth Engine to process large vector layers in parallel.

    -

    This section shows how you can do spatial queries and spatial joins using multiple large feature collections. This requires the use of joins. As described for Image Collections in Chap. F4.9, a join allows you to match every item in a collection with items in another collection based on certain conditions. While you can achieve similar results using map and filter, joins perform better and give you more flexibility. We need to define the following items to perform a join on two collections.

    +

    This section shows how you can do spatial queries and spatial joins using multiple large feature collections. This requires the use of joins. As described for Image Collections in Chap. F4.9, a join allows you to match every item in a collection with items in another collection based on certain conditions. While you can achieve similar results using map and filter, joins perform better and give you more flexibility. We need to define the following items to perform a join on two collections.

      -
    1. Filter: A filter defines the condition used to select the features from the two collections. There is a suite of filters in the ee.Filters module that work on two collections, such as ee.Filter.equals and ee.Filter.withinDistance.
      2. -
    2. Join type: While the filter determines which features will be joined, the join type determines how they will be joined. There are many join types, including simple join, inner join, and save-all join.
      3. +
    3. Filter: A filter defines the condition used to select the features from the two collections. There is a suite of filters in the ee.Filters module that work on two collections, such as ee.Filter.equals and ee.Filter.withinDistance.
      4. +
    4. Join type: While the filter determines which features will be joined, the join type determines how they will be joined. There are many join types, including simple join, inner join, and save-all join.

    Joins are one of the harder skills to master, but doing so will help you perform many complex analysis tasks within Earth Engine. We will go through practical examples that will help you understand these concepts and the workflow better.

    10.2.1 Selecting by Location

    In this section, we will learn how to select features from one layer that are within a specified distance from features in another layer. We will continue to work with the San Francisco census blocks and roads datasets from the previous section. We will implement a join to select all blocks in San Francisco that are within 1 km of an interstate highway.

    We start by loading the census blocks and roads collections and filtering the roads layer to the San Francisco boundary.

    -

    var blocks = ee.FeatureCollection(‘TIGER/2010/Blocks’);
    -var roads = ee.FeatureCollection(‘TIGER/2016/Roads’);
    -var sfNeighborhoods = ee.FeatureCollection(   ‘projects/gee-book/assets/F5-0/SFneighborhoods’);

    -

    var geometry = sfNeighborhoods.geometry();
    +

    var blocks = ee.FeatureCollection(‘TIGER/2010/Blocks’);
    +var roads = ee.FeatureCollection(‘TIGER/2016/Roads’);
    +var sfNeighborhoods = ee.FeatureCollection( ‘projects/gee-book/assets/F5-0/SFneighborhoods’);

    +

    var geometry = sfNeighborhoods.geometry();
    Map.centerObject(geometry);

    -

    // Filter blocks and roads to San Francisco boundary.
    -var sfBlocks = blocks.filter(ee.Filter.bounds(geometry));
    -var sfRoads = roads.filter(ee.Filter.bounds(geometry));

    -

    As we want to select all blocks within 1 km of an interstate highway, we first filter the sfRoads collection to select all segments with the rttyp property value of I.

    -

    var interstateRoads = sfRoads.filter(ee.Filter.eq(‘rttyp’, ‘I’));

    -

    We use the draw function to visualize the sfBlocks and interstateRoads layers (Fig. F5.3.5).

    -

    var sfBlocksDrawn = sfBlocks.draw({
    -       color: ‘gray’,
    -       strokeWidth: 1   })
    -   .clip(geometry);
    +

    // Filter blocks and roads to San Francisco boundary.  
+var sfBlocks = blocks.filter(ee.Filter.bounds(geometry));  
+var sfRoads = roads.filter(ee.Filter.bounds(geometry));
    +

    As we want to select all blocks within 1 km of an interstate highway, we first filter the sfRoads collection to select all segments with the rttyp property value of I.

    +

    var interstateRoads = sfRoads.filter(ee.Filter.eq(‘rttyp’, ‘I’));

    +

    We use the draw function to visualize the sfBlocks and interstateRoads layers (Fig. F5.3.5).

    +

    var sfBlocksDrawn = sfBlocks.draw({
    +color: ‘gray’,
    +strokeWidth: 1 })
    +.clip(geometry);
    Map.addLayer(sfBlocksDrawn, {}, ‘All Blocks’);
    -var interstateRoadsDrawn = interstateRoads.draw({
    -       color: ‘blue’,
    -       strokeWidth: 3   })
    -   .clip(geometry);
    +var interstateRoadsDrawn = interstateRoads.draw({
    +color: ‘blue’,
    +strokeWidth: 3 })
    +.clip(geometry);
    Map.addLayer(interstateRoadsDrawn, {}, ‘Interstate Roads’);

    -

    -

    Fig. F5.3.5 San Francisco blocks and interstate highways

    -

    Let’s define a join that will select all the features from the sfBlocks layer that are within 1 km of any feature from the interstateRoads layer. We start by defining a filter using the ee.Filter.withinDistance filter. We want to compare the geometries of features in both layers, so we use a special property called ‘.geo’ to compare the collections. By default, the filter will work with exact distances between the geometries. If your analysis does not require a very precise tolerance of spatial uncertainty, specifying a small non-zero maxError distance value will help speed up the spatial operations. A larger tolerance also helps when testing or debugging code so you can get the result quickly instead of waiting longer for a more precise output.

    -

    var joinFilter = ee.Filter.withinDistance({
    -   distance: 1000,
    -   leftField: ‘.geo’,
    -   rightField: ‘.geo’,
    -   maxError: 10
    +

    +
    +

    +

    Fig. F5.3.5 San Francisco blocks and interstate highways

    +
    +
    +

    Let’s define a join that will select all the features from the sfBlocks layer that are within 1 km of any feature from the interstateRoads layer. We start by defining a filter using the ee.Filter.withinDistance filter. We want to compare the geometries of features in both layers, so we use a special property called ‘.geo’ to compare the collections. By default, the filter will work with exact distances between the geometries. If your analysis does not require a very precise tolerance of spatial uncertainty, specifying a small non-zero maxError distance value will help speed up the spatial operations. A larger tolerance also helps when testing or debugging code so you can get the result quickly instead of waiting longer for a more precise output.

    +

    var joinFilter = ee.Filter.withinDistance({
    +distance: 1000,
    +leftField: ‘.geo’,
    +rightField: ‘.geo’,
    +maxError: 10
    });

    -

    We will use a simple join as we just want features from the first (primary) collection that match the features from the other (secondary) collection.

    -

    var closeBlocks = ee.Join.simple().apply({
    -   primary: sfBlocks,
    -   secondary: interstateRoads,
    -   condition: joinFilter
    +

    We will use a simple join as we just want features from the first (primary) collection that match the features from the other (secondary) collection.

    +

    var closeBlocks = ee.Join.simple().apply({
    +primary: sfBlocks,
    +secondary: interstateRoads,
    +condition: joinFilter
    });

    We can visualize the results in a different color and verify that the join worked as expected (Fig. F5.3.6).

    -

    var closeBlocksDrawn = closeBlocks.draw({
    -       color: ‘orange’,
    -       strokeWidth: 1   })
    -   .clip(geometry);
    +

    var closeBlocksDrawn = closeBlocks.draw({
    +color: ‘orange’,
    +strokeWidth: 1 })
    +.clip(geometry);
    Map.addLayer(closeBlocksDrawn, {}, ‘Blocks within 1km’);

    -

    -

    Fig. F5.3.6 Selected blocks within 1 km of an interstate highway

    +
    +
    +

    +

    Fig. F5.3.6 Selected blocks within 1 km of an interstate highway

    +
    +

    10.2.2 Spatial Joins

    -

    A spatial join allows you to query two collections based on the spatial relationship. We will now implement a spatial join to count points in polygons. We will work with a dataset of tree locations in San Francisco and polygons of neighborhoods to produce a CSV file with the total number of trees in each neighborhood.

    -

    The San Francisco Open Data Portal maintains a street tree map dataset that has a list of street trees with their latitude and longitude. We will also use the San Francisco neighborhood dataset from the same portal. We downloaded, processed, and uploaded these layers as Earth Engine assets for use in this exercise. We start by loading both layers and using the paint and style functions, covered in Sect. 1, to visualize them (Fig. F5.3.7).

    -

    var sfNeighborhoods = ee.FeatureCollection(   ‘projects/gee-book/assets/F5-0/SFneighborhoods’);
    -var sfTrees = ee.FeatureCollection(   ‘projects/gee-book/assets/F5-3/SFTrees’);

    -

    // Use paint() to visualize the polygons with only outline
    -var sfNeighborhoodsOutline = ee.Image().byte().paint({
    -   featureCollection: sfNeighborhoods,
    -   color: 1,
    -   width: 3
    -});
    -Map.addLayer(sfNeighborhoodsOutline, {
    -       palette: [‘blue’]
    -   },   ‘SF Neighborhoods’);

    -

    // Use style() to visualize the points
    -var sfTreesStyled = sfTrees.style({
    -   color: ‘green’,
    -   pointSize: 2,
    -   pointShape: ‘triangle’,
    -   width: 2
    -});
    -Map.addLayer(sfTreesStyled, {}, ‘SF Trees’);

    -

    -

    Fig. F5.3.7 San Francisco neighborhoods and trees

    -

    To find the tree points in each neighborhood polygon, we will use an ee.Filter.intersects filter.

    -

    var intersectFilter = ee.Filter.intersects({
    -   leftField: ‘.geo’,
    -   rightField: ‘.geo’,
    -   maxError: 10
    +

    A spatial join allows you to query two collections based on the spatial relationship. We will now implement a spatial join to count points in polygons. We will work with a dataset of tree locations in San Francisco and polygons of neighborhoods to produce a CSV file with the total number of trees in each neighborhood.

    +

    The San Francisco Open Data Portal maintains a street tree map dataset that has a list of street trees with their latitude and longitude. We will also use the San Francisco neighborhood dataset from the same portal. We downloaded, processed, and uploaded these layers as Earth Engine assets for use in this exercise. We start by loading both layers and using the paint and style functions, covered in Sect. 1, to visualize them (Fig. F5.3.7).

    +

    var sfNeighborhoods = ee.FeatureCollection( ‘projects/gee-book/assets/F5-0/SFneighborhoods’);
    +var sfTrees = ee.FeatureCollection( ‘projects/gee-book/assets/F5-3/SFTrees’);

    +
    // Use paint() to visualize the polygons with only outline  
+var sfNeighborhoodsOutline = ee.Image().byte().paint({  
+   featureCollection: sfNeighborhoods,  
+   color: 1,  
+   width: 3  
+});  
+Map.addLayer(sfNeighborhoodsOutline, {  
+       palette: ['blue']  
+   },   'SF Neighborhoods');  
+  
+// Use style() to visualize the points  
+var sfTreesStyled = sfTrees.style({  
+   color: 'green',  
+   pointSize: 2,  
+   pointShape: 'triangle',  
+   width: 2  
+});  
+Map.addLayer(sfTreesStyled, {}, 'SF Trees');
    +
    +
    +

    +

    Fig. F5.3.7 San Francisco neighborhoods and trees

    +
    +
    +

    To find the tree points in each neighborhood polygon, we will use an ee.Filter.intersects filter.

    +

    var intersectFilter = ee.Filter.intersects({
    +leftField: ‘.geo’,
    +rightField: ‘.geo’,
    +maxError: 10
    });

    -

    We need a join that can give us a list of all tree features that intersect each neighborhood polygon, so we need to use a saving join. A saving join will find all the features from the secondary collection that match the filter and store them in a property in the primary collection. Once you apply this join, you will get a version of the primary collection with an additional property that has the matching features from the secondary collection. Here we use the ee.Join.saveAll join, since we want to store all matching features. We specify the matchesKey property that will be added to each feature with the results.

    -

    var saveAllJoin = ee.Join.saveAll({
    -   matchesKey: ‘trees’,
    +

    We need a join that can give us a list of all tree features that intersect each neighborhood polygon, so we need to use a saving join. A saving join will find all the features from the secondary collection that match the filter and store them in a property in the primary collection. Once you apply this join, you will get a version of the primary collection with an additional property that has the matching features from the secondary collection. Here we use the ee.Join.saveAll join, since we want to store all matching features. We specify the matchesKey property that will be added to each feature with the results.

    +

    var saveAllJoin = ee.Join.saveAll({
    +matchesKey: ‘trees’,
    });

    Let’s apply the join and print the first feature of the resulting collection to verify (Fig. F5.3.8).

    -

    var joined = saveAllJoin
    -   .apply(sfNeighborhoods, sfTrees, intersectFilter);
    +

    var joined = saveAllJoin
    +.apply(sfNeighborhoods, sfTrees, intersectFilter);
    print(joined.first());

    -

    -

    Fig. F5.3.8 Result of the save-all join

    -

    You will see that each feature of the sfNeighborhoods collection now has an additional property called trees. This contains all the features from the sfTrees collection that were matched using the intersectFilter. We can now map a function over the results and post-process the collection. As our analysis requires the computation of the total number of trees in each neighborhood, we extract the matching features and use the size function to get the count (Fig. F5.3.9).

    -

    // Calculate total number of trees within each feature.
    -var sfNeighborhoods = joined.map(function(f) {   var treesWithin = ee.List(f.get(‘trees’));   var totalTrees = ee.FeatureCollection(treesWithin).size();   return f.set(‘total_trees’, totalTrees);
    -});

    -

    print(sfNeighborhoods.first());

    -

    -

    Fig. F5.3.9 Final FeatureCollection with the new property

    -

    The results now have a property called total_trees containing the count of intersecting trees in each neighborhood polygon.

    -

    The final step in the analysis is to export the results as a CSV file using the Export.table.toDrive function. Note that as described in detail in F6.2, you should output only the columns you need to the CSV file. Suppose we do not need all the properties to appear in the output; imagine that wedo not need the trees property, for example, in the output. In that case, we can create only those columns we want in the manner below, by specifying the other selectors parameters with the list of properties to export.

    -

    // Export the results as a CSV.
    -Export.table.toDrive({
    -   collection: sfNeighborhoods,
    -   description: ‘SF_Neighborhood_Tree_Count’,
    -   folder: ‘earthengine’,
    -   fileNamePrefix: ‘tree_count’,
    -   fileFormat: ‘CSV’,
    -   selectors: [‘nhood’, ‘total_trees’]
    -});

    +
    +
    +

    +

    Fig. F5.3.8 Result of the save-all join

    +
    +
    +

    You will see that each feature of the sfNeighborhoods collection now has an additional property called trees. This contains all the features from the sfTrees collection that were matched using the intersectFilter. We can now map a function over the results and post-process the collection. As our analysis requires the computation of the total number of trees in each neighborhood, we extract the matching features and use the size function to get the count (Fig. F5.3.9).

    +
    // Calculate total number of trees within each feature.  
+var sfNeighborhoods = joined.map(function(f) {   var treesWithin = ee.List(f.get('trees'));   var totalTrees = ee.FeatureCollection(treesWithin).size();   return f.set('total_trees', totalTrees);  
+});  
+  
+print(sfNeighborhoods.first());
    +
    +
    +

    +

    Fig. F5.3.9 Final FeatureCollection with the new property

    +
    +
    +

    The results now have a property called total_trees containing the count of intersecting trees in each neighborhood polygon.

    +

    The final step in the analysis is to export the results as a CSV file using the Export.table.toDrive function. Note that as described in detail in F6.2, you should output only the columns you need to the CSV file. Suppose we do not need all the properties to appear in the output; imagine that wedo not need the trees property, for example, in the output. In that case, we can create only those columns we want in the manner below, by specifying the other selectors parameters with the list of properties to export.

    +
    // Export the results as a CSV.  
+Export.table.toDrive({  
+   collection: sfNeighborhoods,  
+   description: 'SF_Neighborhood_Tree_Count',  
+   folder: 'earthengine',  
+   fileNamePrefix: 'tree_count',  
+   fileFormat: 'CSV',  
+   selectors: ['nhood', 'total_trees']  
+});

    The final result is a CSV file with the neighborhood names and total numbers of trees counted using the join (Fig. F5.3.10).

    -

    -

    Fig. F5.3.10 Exported CSV file with tree counts for San Francisco neighborhoods

    +
    +
    +

    +

    Fig. F5.3.10 Exported CSV file with tree counts for San Francisco neighborhoods

    +
    +
    @@ -1971,18 +2222,18 @@ Note
    -

    Code Checkpoint F53b. The book’s repository contains a script that shows what your code should look like at this point.

    +

    Code Checkpoint F53b. The book’s repository contains a script that shows what your code should look like at this point.

    Synthesis

    -

    Assignment 1. What join would you use if you wanted to know which neighborhood each tree belongs to? Modify the code above to do a join and post-process the result to add a neighborhood property to each tree point. Export the results as a shapefile.

    +

    Assignment 1. What join would you use if you wanted to know which neighborhood each tree belongs to? Modify the code above to do a join and post-process the result to add a neighborhood property to each tree point. Export the results as a shapefile.

    Conclusion

    -

    This chapter covered visualization and analysis using vector data in Earth Engine. You should now understand different functions for FeatureCollection visualization and be able to create thematic maps with vector layers. You also learned techniques for doing spatial queries and spatial joins within Earth Engine. Earth Engine is capable of handling large feature collections and can be effectively used for many spatial analysis tasks.

    +

    This chapter covered visualization and analysis using vector data in Earth Engine. You should now understand different functions for FeatureCollection visualization and be able to create thematic maps with vector layers. You also learned techniques for doing spatial queries and spatial joins within Earth Engine. Earth Engine is capable of handling large feature collections and can be effectively used for many spatial analysis tasks.

    diff --git a/docs/F6.html b/docs/F6.html index cae7ff6..3012ae9 100644 --- a/docs/F6.html +++ b/docs/F6.html @@ -20,6 +20,69 @@ ul.task-list li input[type="checkbox"] { margin: 0 0.8em 0.2em -1.6em; vertical-align: middle; } +pre > code.sourceCode { white-space: pre; position: relative; } +pre > code.sourceCode > span { display: inline-block; line-height: 1.25; } +pre > code.sourceCode > span:empty { height: 1.2em; } +.sourceCode { overflow: visible; } +code.sourceCode > span { color: inherit; text-decoration: inherit; } +div.sourceCode { margin: 1em 0; } +pre.sourceCode { margin: 0; } +@media screen { +div.sourceCode { overflow: auto; } +} +@media print { +pre > code.sourceCode { white-space: pre-wrap; } +pre > code.sourceCode > span { text-indent: -5em; padding-left: 5em; } +} +pre.numberSource code + { counter-reset: source-line 0; } +pre.numberSource code > span + { position: relative; left: -4em; counter-increment: source-line; } +pre.numberSource code > span > a:first-child::before + { content: counter(source-line); + position: relative; left: -1em; text-align: right; vertical-align: baseline; + border: none; display: inline-block; + -webkit-touch-callout: none; -webkit-user-select: none; + -khtml-user-select: none; -moz-user-select: none; + -ms-user-select: none; user-select: none; + padding: 0 4px; width: 4em; + color: #aaaaaa; + } +pre.numberSource { margin-left: 3em; border-left: 1px solid #aaaaaa; padding-left: 4px; } +div.sourceCode + { } +@media screen { +pre > code.sourceCode > span > a:first-child::before { text-decoration: underline; } +} +code span.al { color: #ff0000; font-weight: bold; } /* Alert */ +code span.an { color: #60a0b0; font-weight: bold; font-style: italic; } /* Annotation */ +code span.at { color: #7d9029; } /* Attribute */ +code span.bn { color: #40a070; } /* BaseN */ +code span.bu { color: #008000; } /* BuiltIn */ +code span.cf { color: #007020; font-weight: bold; } /* ControlFlow */ +code span.ch { color: #4070a0; } /* Char */ +code span.cn { color: #880000; } /* Constant */ +code span.co { color: #60a0b0; font-style: italic; } /* Comment */ +code span.cv { color: #60a0b0; font-weight: bold; font-style: italic; } /* CommentVar */ +code span.do { color: #ba2121; font-style: italic; } /* Documentation */ +code span.dt { color: #902000; } /* DataType */ +code span.dv { color: #40a070; } /* DecVal */ +code span.er { color: #ff0000; font-weight: bold; } /* Error */ +code span.ex { } /* Extension */ +code span.fl { color: #40a070; } /* Float */ +code span.fu { color: #06287e; } /* Function */ +code span.im { color: #008000; font-weight: bold; } /* Import */ +code span.in { color: #60a0b0; font-weight: bold; font-style: italic; } /* Information */ +code span.kw { color: #007020; font-weight: bold; } /* Keyword */ +code span.op { color: #666666; } /* Operator */ +code span.ot { color: #007020; } /* Other */ +code span.pp { color: #bc7a00; } /* Preprocessor */ +code span.sc { color: #4070a0; } /* SpecialChar */ +code span.ss { color: #bb6688; } /* SpecialString */ +code span.st { color: #4070a0; } /* String */ +code span.va { color: #19177c; } /* Variable */ +code span.vs { color: #4070a0; } /* VerbatimString */ +code span.wa { color: #60a0b0; font-weight: bold; font-style: italic; } /* Warning */ @@ -231,7 +294,7 @@ gtag('config', 'G-RK9ZLZQ6GL', { 'anonymize_ip': true});
  • 8 Advanced Raster Visualization
    • -
  • 9 Collaborating in Earth Engine with Scripts and Assets +
  • 9 Collaborating in Earth Engine with Scripts and Assets
    • @@ -286,20 +345,24 @@ gtag('config', 'G-RK9ZLZQ6GL', { 'anonymize_ip': true});
  • 18 Use install.packages to install R packages from the CRAN repository.
  • 19 Initialize just Earth Engine
  • 20 Initialize Earth Engine and GDee_Initialize(drive = TRUE)
  • 21 Define a ImageCollection
  • 22 Set viz paramsviz <- list(
    • -
  • 23 Print map results interactively
    • +
  • 23 Print map results interactively +
  • 24 Estimate the median of the yearly composites from 2016 to 2020.
  • 25 Estimate the median of the winter season composites from 2016 to 2020.
      +
    • 25.1 Converting JavaScript Modules to R
    • Synthesis
    • Conclusion
    • References
      • @@ -362,8 +425,8 @@ Chapter Information

      Assumes you know how to:

      • Import images and image collections, filter, and visualize (Part F1).
        • -
      • Write a function and map it over an ImageCollection (Chap. F4.0).
        • -
      • Inspect an Image and an ImageCollection, as well as their properties (Chap. F4.1).
        • +
      • Write a function and map it over an ImageCollection (Chap. F4.0).
        • +
      • Inspect an Image and an ImageCollection, as well as their properties (Chap. F4.1).
      @@ -387,50 +450,60 @@ Chapter Information

      8.1 Palettes

      In this section we will explore examples of colormaps to visualize raster data. Colormaps translate values to colors for display on a map. This requires a set of colors (referred to as a “palette” in Earth Engine) and a range of values to map (specified by the min and max values in the visualization parameters).

      There are multiple types of colormaps, each used for a different purpose. These include the following:

      -

      Sequential: These are probably the most commonly used colormaps, and are useful for ordinal, interval, and ratio data. Also referred to as a linear colormap, a sequential colormap looks like the viridis colormap (Fig. F6.0.1) from matplotlib. It is popular because it is a perceptual uniform colormap, where an equal interval in values is mapped to an equal interval in the perceptual colorspace. If you have a ratio variable where zero means nothing, you can use a sequential colormap starting at white, transparent, or, when you have a black background, at black—for example, the turku colormap from Crameri (Fig. F6.0.1). You can use this for variables like population count or gross domestic product.

      -

      Diverging: This type of colormap is used for visualizing data where you have positive and negative values and where zero has a meaning. Later in this tutorial, we will use the balance colormap from the cmocean package (Fig. F6.0.1) to show temperature change.

      -

      Circular: Some variables are periodic, returning to the same value after a period of time. For example, the season, angle, and time of day are typically represented as circular variables. For variables like this, a circular colormap is designed to represent the first and last values with the same color. An example is the circular cet-c2 colormap (Fig. F6.0.1) from the colorcet package.

      -

      Semantic: Some colormaps do not map to arbitrary colors but choose colors that provide meaning. We refer to these as semantic colormaps. Later in this tutorial, we will use the ice colormap (Fig. F6.0.1) from the cmocean package for our ice example.

      -

      -

      Fig. F6.0.1 Examples of colormaps from a variety of packages: viridis from matplotlib, turku from Crameri, balance from cmocean, cet-c2 from colorcet and ice from cmocean

      +

      Sequential: These are probably the most commonly used colormaps, and are useful for ordinal, interval, and ratio data. Also referred to as a linear colormap, a sequential colormap looks like the viridis colormap (Fig. F6.0.1) from matplotlib. It is popular because it is a perceptual uniform colormap, where an equal interval in values is mapped to an equal interval in the perceptual colorspace. If you have a ratio variable where zero means nothing, you can use a sequential colormap starting at white, transparent, or, when you have a black background, at black—for example, the turku colormap from Crameri (Fig. F6.0.1). You can use this for variables like population count or gross domestic product.

      +

      Diverging: This type of colormap is used for visualizing data where you have positive and negative values and where zero has a meaning. Later in this tutorial, we will use the balance colormap from the cmocean package (Fig. F6.0.1) to show temperature change.

      +

      Circular: Some variables are periodic, returning to the same value after a period of time. For example, the season, angle, and time of day are typically represented as circular variables. For variables like this, a circular colormap is designed to represent the first and last values with the same color. An example is the circular cet-c2 colormap (Fig. F6.0.1) from the colorcet package.

      +

      Semantic: Some colormaps do not map to arbitrary colors but choose colors that provide meaning. We refer to these as semantic colormaps. Later in this tutorial, we will use the ice colormap (Fig. F6.0.1) from the cmocean package for our ice example.

      +
      +
      +

      +

      Fig. F6.0.1 Examples of colormaps from a variety of packages: viridis from matplotlib, turku from Crameri, balance from cmocean, cet-c2 from colorcet and ice from cmocean

      +
      +

      Popular sources of colormaps include:

        -
      • cmocean (semantic perceptual uniform colormaps for geophysical applications)
        • +
      • cmocean (semantic perceptual uniform colormaps for geophysical applications)
      • colorcet (set of perceptual colormaps with varying colors and saturation)
      • cpt-city (comprehensive overview of colormaps,
      • colorbrewer (colormaps with variety of colors)
      • Crameri (stylish colormaps for dark and light themes)
      -

      Our first example in this section applies a diverging colormap to temperature.

      -

      // Load the ERA5 reanalysis monthly means.
      -var era5 = ee.ImageCollection(‘ECMWF/ERA5_LAND/MONTHLY’);

      -

      // Load the palettes package.
      -var palettes = require(‘users/gena/packages:palettes’);

      -

      // Select temperature near ground.
      -era5 = era5.select(‘temperature_2m’);

      -

      Now we can visualize the data. Here we have a temperature difference. That means that zero has a special meaning. By using a divergent colormap we can give zero the color white, which denotes that there is no significant difference. Here we will use the colormap Balance from the cmocean package. The color red is associated with warmth, and the color blue is associated with cold. We will choose the minimum and maximum values for the palette to be symmetric around zero (-2, 2) so that white appears in the correct place.  For comparison we also visualize the data with a simple [‘blue’, ‘white’, ‘red’] palette. As you can see (Fig. F6.0.2), the Balance colormap has a more elegant and professional feel to it, because it uses a perceptual uniform palette and both saturation and value.

      -

      // Choose a diverging colormap for anomalies.
      -var balancePalette = palettes.cmocean.Balance[7];
      -var threeColorPalette = [‘blue’, ‘white’, ‘red’];

      -

      // Show the palette in the Inspector window.
      -palettes.showPalette(‘temperature anomaly’, balancePalette);
      -palettes.showPalette(‘temperature anomaly’, threeColorPalette);

      -

      // Select 2 time windows of 10 years.
      -var era5_1980 = era5.filterDate(‘1981-01-01’, ‘1991-01-01’).mean();
      -var era5_2010 = era5.filterDate(‘2011-01-01’, ‘2020-01-01’).mean();

      -

      // Compute the temperature change.
      -var era5_diff = era5_2010.subtract(era5_1980);

      -

      // Show it on the map.
      -Map.addLayer(era5_diff, {
      -   palette: threeColorPalette,
      -   min: -2,
      -   max: 2}, ‘Blue White Red palette’);

      -

      Map.addLayer(era5_diff, {
      -   palette: balancePalette,
      -   min: -2,
      -   max: 2}, ‘Balance palette’);

      -

      -

      Fig. F6.0.2 Temperature difference of ERA5 (2011–2020, 1981–1990) using the balance colormap from cmocean (right) versus a basic blue-white-red colormap (left)

      +

      Our first example in this section applies a diverging colormap to temperature.

      +
      // Load the ERA5 reanalysis monthly means.  
+var era5 = ee.ImageCollection('ECMWF/ERA5_LAND/MONTHLY');  
+  
+// Load the palettes package.  
+var palettes = require('users/gena/packages:palettes');  
+  
+// Select temperature near ground.  
+era5 = era5.select('temperature_2m');
      +

      Now we can visualize the data. Here we have a temperature difference. That means that zero has a special meaning. By using a divergent colormap we can give zero the color white, which denotes that there is no significant difference. Here we will use the colormap Balance from the cmocean package. The color red is associated with warmth, and the color blue is associated with cold. We will choose the minimum and maximum values for the palette to be symmetric around zero (-2, 2) so that white appears in the correct place. For comparison we also visualize the data with a simple [‘blue’, ‘white’, ‘red’] palette. As you can see (Fig. F6.0.2), the Balance colormap has a more elegant and professional feel to it, because it uses a perceptual uniform palette and both saturation and value.

      +
      // Choose a diverging colormap for anomalies.  
+var balancePalette = palettes.cmocean.Balance[7];  
+var threeColorPalette = ['blue', 'white', 'red'];  
+  
+// Show the palette in the Inspector window.  
+palettes.showPalette('temperature anomaly', balancePalette);  
+palettes.showPalette('temperature anomaly', threeColorPalette);  
+  
+// Select 2 time windows of 10 years.  
+var era5_1980 = era5.filterDate('1981-01-01', '1991-01-01').mean();  
+var era5_2010 = era5.filterDate('2011-01-01', '2020-01-01').mean();  
+  
+// Compute the temperature change.  
+var era5_diff = era5_2010.subtract(era5_1980);  
+  
+// Show it on the map.  
+Map.addLayer(era5_diff, {  
+   palette: threeColorPalette,  
+   min: -2,  
+   max: 2}, 'Blue White Red palette');  
+  
+Map.addLayer(era5_diff, {  
+   palette: balancePalette,  
+   min: -2,  
+   max: 2}, 'Balance palette');
      +

      Fig. F6.0.2 Temperature difference of ERA5 (2011–2020, 1981–1990) using the balance colormap from cmocean (right) versus a basic blue-white-red colormap (left)Fig. F6.0.2 Temperature difference of ERA5 (2011–2020, 1981–1990) using the balance colormap from cmocean (right) versus a basic blue-white-red colormap (left)

      @@ -444,29 +517,38 @@ Note

      Code Checkpoint F60a. The book’s repository contains a script that shows what your code should look like at this point.

      -

      Our second example in this section focuses on visualizing a region of the Antarctic, the Thwaites Glacier. This is one of the fast-flowing glaciers that causes concern because it loses so much mass that it causes the sea level to rise. If we want to visualize this region, we have a challenge. The Antarctic region is in the dark for four to five months each winter. That means that we can’t use optical images to see the ice flowing into the sea. We therefore will use radar images. Here we will use a semantic colormap to denote the meaning of the radar images.

      +

      Our second example in this section focuses on visualizing a region of the Antarctic, the Thwaites Glacier. This is one of the fast-flowing glaciers that causes concern because it loses so much mass that it causes the sea level to rise. If we want to visualize this region, we have a challenge. The Antarctic region is in the dark for four to five months each winter. That means that we can’t use optical images to see the ice flowing into the sea. We therefore will use radar images. Here we will use a semantic colormap to denote the meaning of the radar images.

      Let’s start by importing the dataset of radar images. We will use the images from the Sentinel-1 constellation of the Copernicus program. This satellite uses a C-band synthetic-aperture radar and has near-polar coverage. The radar senses images using a polarity for the sender and receiver. The collection has images of four different possible combinations of sender/receiver polarity pairs. The image that we’ll use has a band of the Horizontal/Horizontal polarity (HH).

      -

      // An image of the Thwaites glacier.
      -var imageId =‘COPERNICUS/S1_GRD/S1B_EW_GRDM_1SSH_20211216T041925_20211216T042029_030045_03965B_AF0A’;

      -

      // Look it up and select the HH band.
      -var img = ee.Image(imageId).select(‘HH’);

      -

      For the next step, we will use the palette library. We will stylize the radar images to look like optical images, so that viewers can contrast ice and sea ice from water (Lhermitte, 2020). We will use the Ice colormap from the cmocean package (Thyng, 2016).

      -

      // Use the palette library.
      -var palettes = require(‘users/gena/packages:palettes’);

      -

      // Access the ice palette.
      -var icePalette = palettes.cmocean.Ice[7];

      -

      // Show it in the console.
      -palettes.showPalette(‘Ice’, icePalette);

      -

      // Use  it to visualize the radar data.
      -Map.addLayer(img, {
      -   palette: icePalette,
      -   min: -15,
      -   max: 1}, ‘Sentinel-1 radar’);

      -

      // Zoom to the grounding line of the Thwaites Glacier.
      -Map.centerObject(ee.Geometry.Point([-105.45882094907664, -   74.90419580705336]), 8);

      -

      If you zoom in (F6.0.3) you can see how long cracks have recently appeared near the pinning point (a peak in the bathymetry that functions as a buttress, see Wild, 2022) of the glacier.

      -

      -

      Fig. F6.0.3. Ice observed in Antarctica by the Sentinel-1 satellite. The image is rendered using the ice color palette stretched to backscatter amplitude values [-15; 1].

      +
      // An image of the Thwaites glacier.  
+var imageId ='COPERNICUS/S1_GRD/S1B_EW_GRDM_1SSH_20211216T041925_20211216T042029_030045_03965B_AF0A';  
+  
+// Look it up and select the HH band.  
+var img = ee.Image(imageId).select('HH');
      +

      For the next step, we will use the palette library. We will stylize the radar images to look like optical images, so that viewers can contrast ice and sea ice from water (Lhermitte, 2020). We will use the Ice colormap from the cmocean package (Thyng, 2016).

      +
      // Use the palette library.  
+var palettes = require('users/gena/packages:palettes');  
+  
+// Access the ice palette.  
+var icePalette = palettes.cmocean.Ice[7];  
+  
+// Show it in the console.  
+palettes.showPalette('Ice', icePalette);  
+  
+// Use  it to visualize the radar data.  
+Map.addLayer(img, {  
+   palette: icePalette,  
+   min: -15,  
+   max: 1}, 'Sentinel-1 radar');  
+  
+// Zoom to the grounding line of the Thwaites Glacier.  
+Map.centerObject(ee.Geometry.Point([-105.45882094907664, -   74.90419580705336]), 8);
      +

      If you zoom in (F6.0.3) you can see how long cracks have recently appeared near the pinning point (a peak in the bathymetry that functions as a buttress, see Wild, 2022) of the glacier.

      +
      +
      +

      +

      Fig. F6.0.3. Ice observed in Antarctica by the Sentinel-1 satellite. The image is rendered using the ice color palette stretched to backscatter amplitude values [-15; 1].

      +
      +
      @@ -482,53 +564,74 @@ Note
      -

      8.2 Remapping and Palettes

      -

      Classified rasters in Earth Engine have metadata attached that can help with analysis and visualization. This includes lists of the names, values, and colors associated with class. These are used as the default color palette for drawing a classification, as seen next. The USGS National Land Cover Database (NLCD) is one such example. Let’s access the NLCD dataset, name it nlcd, and view it (Fig. F6.0.4) with its built-in palette.

      -

      // Advanced remapping using NLCD.
      -// Import NLCD.
      -var nlcd = ee.ImageCollection(‘USGS/NLCD_RELEASES/2016_REL’);

      -

      // Use Filter to select the 2016 dataset.
      -var nlcd2016 = nlcd.filter(ee.Filter.eq(‘system:index’, ‘2016’))
      -   .first();

      -

      // Select the land cover band.
      -var landcover = nlcd2016.select(‘landcover’);

      -

      // Map the NLCD land cover.
      -Map.addLayer(landcover, null, ‘NLCD Landcover’);

      -

      -

      Fig. F6.0.4 The NLCD visualized with default colors for each class

      +

      8.2 Remapping and Palettes

      +

      Classified rasters in Earth Engine have metadata attached that can help with analysis and visualization. This includes lists of the names, values, and colors associated with class. These are used as the default color palette for drawing a classification, as seen next. The USGS National Land Cover Database (NLCD) is one such example. Let’s access the NLCD dataset, name it nlcd, and view it (Fig. F6.0.4) with its built-in palette.

      +
      // Advanced remapping using NLCD.  
+// Import NLCD.  
+var nlcd = ee.ImageCollection('USGS/NLCD_RELEASES/2016_REL');  
+  
+// Use Filter to select the 2016 dataset.  
+var nlcd2016 = nlcd.filter(ee.Filter.eq('system:index', '2016'))  
+   .first();  
+  
+// Select the land cover band.  
+var landcover = nlcd2016.select('landcover');  
+  
+// Map the NLCD land cover.  
+Map.addLayer(landcover, null, 'NLCD Landcover');
      +
      +
      +

      +

      Fig. F6.0.4 The NLCD visualized with default colors for each class

      +
      +

      But suppose you want to change the display palette. For example, you might want to have multiple classes displayed using the same color, or use different colors for some classes. Let’s try having all three urban classes display as dark red (‘ab0000’).

      -

      // Now suppose we want to change the color palette.
      -var newPalette = [‘466b9f’, ‘d1def8’, ‘dec5c5’,   ‘ab0000’, ‘ab0000’, ‘ab0000’,   ‘b3ac9f’, ‘68ab5f’, ‘1c5f2c’,   ‘b5c58f’, ‘af963c’, ‘ccb879’,   ‘dfdfc2’, ‘d1d182’, ‘a3cc51’,   ‘82ba9e’, ‘dcd939’, ‘ab6c28’,   ‘b8d9eb’, ‘6c9fb8’
      -];

      -

      // Try mapping with the new color palette.
      -Map.addLayer(landcover, {
      -   palette: newPalette
      -}, ‘NLCD New Palette’);

      +
      // Now suppose we want to change the color palette.  
+var newPalette = ['466b9f', 'd1def8', 'dec5c5',   'ab0000', 'ab0000', 'ab0000',   'b3ac9f', '68ab5f', '1c5f2c',   'b5c58f', 'af963c', 'ccb879',   'dfdfc2', 'd1d182', 'a3cc51',   '82ba9e', 'dcd939', 'ab6c28',   'b8d9eb', '6c9fb8'  
+];  
+  
+// Try mapping with the new color palette.  
+Map.addLayer(landcover, {  
+   palette: newPalette  
+}, 'NLCD New Palette');

      However, if you map this, you will see an unexpected result (Fig. F6.0.5).

      -

      -

      Fig. F6.0.5 Applying a new palette to a multi-class layer has some unexpected results

      +
      +
      +

      +

      Fig. F6.0.5 Applying a new palette to a multi-class layer has some unexpected results

      +
      +

      This is because the numeric codes for the different classes are not sequential. Thus, Earth Engine stretches the given palette across the whole range of values and produces an unexpected color palette. To fix this issue, we will create a new index for the class values so that they are sequential.

      -

      // Extract the class values and save them as a list.
      -var values = ee.List(landcover.get(‘landcover_class_values’));

      -

      // Print the class values to console.
      -print(‘raw class values’, values);

      -

      // Determine the maximum index value
      -var maxIndex = values.size().subtract(1);

      -

      // Create a new index for the remap
      -var indexes = ee.List.sequence(0, maxIndex);

      -

      // Print the updated class values to console.
      -print(‘updated class values’, indexes);

      -

      // Remap NLCD and display it in the map.
      -var colorized = landcover.remap(values, indexes)
      -   .visualize({
      -       min: 0,
      -       max: maxIndex,
      -       palette: newPalette
      -   });
      -Map.addLayer(colorized, {}, ‘NLCD Remapped Colors’);

      +
      // Extract the class values and save them as a list.  
+var values = ee.List(landcover.get('landcover_class_values'));  
+  
+// Print the class values to console.  
+print('raw class values', values);  
+  
+// Determine the maximum index value  
+var maxIndex = values.size().subtract(1);  
+  
+// Create a new index for the remap  
+var indexes = ee.List.sequence(0, maxIndex);  
+  
+// Print the updated class values to console.  
+print('updated class values', indexes);  
+  
+// Remap NLCD and display it in the map.  
+var colorized = landcover.remap(values, indexes)  
+   .visualize({  
+       min: 0,  
+       max: maxIndex,  
+       palette: newPalette  
+   });  
+Map.addLayer(colorized, {}, 'NLCD Remapped Colors');

      Using this remapping approach, we can properly visualize the new color palette (Fig. F6.0.6).

      -

      -

      Fig. F6.0.6 Expected results of the new color palette. All urban areas are now correctly showing as dark red and the other land cover types remain their original color.

      +
      +
      +

      +

      Fig. F6.0.6 Expected results of the new color palette. All urban areas are now correctly showing as dark red and the other land cover types remain their original color.

      +
      +
      @@ -545,10 +648,10 @@ Note

      8.3 Annotations

      -

      Annotations are the way to visualize data on maps to provide additional information about raster values or any other data relevant to the context. In this case, this additional information is usually shown as geometries, text labels, diagrams, or other visual elements. Some annotations in Earth Engine can be added by making use of the ui portion of the Earth Engine API, resulting in graphical user interface elements such as labels or charts added on top of the map. However, it is frequently useful to render annotations as a part of images, such as by visualizing various image properties or to highlight specific areas.

      -

      In many cases, these annotations can be mixed with output images generated outside of Earth Engine, for example, by post-processing exported images using Python libraries or by annotating using GIS applications such as QGIS or ArcGIS. However, annotations could also be also very useful to highlight and/or label specific areas directly within the Code Editor. Earth Engine provides a sufficiently rich API to turn vector features and geometries into raster images which can serve as annotations. We recommend checking the ee.FeatureCollection.style function in the Earth Engine documentation to learn how geometries can be rendered.

      -

      For  textual annotation, we will make use of an external package ‘users/gena/packages:text’ that provides a way to render strings into raster images directly using the Earth Engine raster API. It is beyond the scope of the current tutorials to explain the implementation of this package, but internally this package makes use of bitmap fonts which are ingested into Earth Engine as raster assets and are used to turn every character of a provided string into image glyphs, which are then translated to desired coordinates.

      -

      The API of the text package includes the following mandatory and optional arguments:

      +

      Annotations are the way to visualize data on maps to provide additional information about raster values or any other data relevant to the context. In this case, this additional information is usually shown as geometries, text labels, diagrams, or other visual elements. Some annotations in Earth Engine can be added by making use of the ui portion of the Earth Engine API, resulting in graphical user interface elements such as labels or charts added on top of the map. However, it is frequently useful to render annotations as a part of images, such as by visualizing various image properties or to highlight specific areas.

      +

      In many cases, these annotations can be mixed with output images generated outside of Earth Engine, for example, by post-processing exported images using Python libraries or by annotating using GIS applications such as QGIS or ArcGIS. However, annotations could also be also very useful to highlight and/or label specific areas directly within the Code Editor. Earth Engine provides a sufficiently rich API to turn vector features and geometries into raster images which can serve as annotations. We recommend checking the ee.FeatureCollection.style function in the Earth Engine documentation to learn how geometries can be rendered.

      +

      For textual annotation, we will make use of an external package ‘users/gena/packages:text’ that provides a way to render strings into raster images directly using the Earth Engine raster API. It is beyond the scope of the current tutorials to explain the implementation of this package, but internally this package makes use of bitmap fonts which are ingested into Earth Engine as raster assets and are used to turn every character of a provided string into image glyphs, which are then translated to desired coordinates.

      +

      The API of the text package includes the following mandatory and optional arguments:

      /**
      * Draws a string as a raster image at a given point.
      *
      @@ -557,29 +660,31 @@ Note * (param?) {{string, Object}} options - optional properties used to style text
      *
      * The options dictionary may include one or more of the following:
      -*     fontSize       - 16|18|24|32 - the size of the font (default: 16)
      -*     fontType       - Arial|Consolas - the type of the font (default: Arial)
      -*     alignX         - left|center|right (default: left)
      -*     alignY         - top|center|bottom (default: top)
      -*     textColor      - text color string (default: ffffff - white)
      -*     textOpacity    - 0-1, opacity of the text (default: 0.9)
      -*     textWidth      - width of the text (default: 1)
      -*     outlineColor   - text outline color string (default: 000000 - black)
      -*     outlineOpacity - 0-1, opacity of the text outline (default: 0.4)
      -*     outlineWidth   - width of the text outlines (default: 0)
      +* fontSize - 16|18|24|32 - the size of the font (default: 16)
      +* fontType - Arial|Consolas - the type of the font (default: Arial)
      +* alignX - left|center|right (default: left)
      +* alignY - top|center|bottom (default: top)
      +* textColor - text color string (default: ffffff - white)
      +* textOpacity - 0-1, opacity of the text (default: 0.9)
      +* textWidth - width of the text (default: 1)
      +* outlineColor - text outline color string (default: 000000 - black)
      +* outlineOpacity - 0-1, opacity of the text outline (default: 0.4)
      +* outlineWidth - width of the text outlines (default: 0)
      */

      -

      To demonstrate how to use this API, let’s render a simple ‘Hello World!’ text string placed at the map center using default text parameters. The code for this will be:

      -

      // Include the text package.
      -var text = require(‘users/gena/packages:text’);

      -

      // Configure map (change center and map type).
      -Map.setCenter(0, 0, 10);
      -Map.setOptions(‘HYBRID’);

      -

      // Draw text string and add to map.
      -var pt = Map.getCenter();
      -var scale = Map.getScale();
      -var image = text.draw(‘Hello World!’, pt, scale);
      -Map.addLayer(image);

      -

      Running the above script will generate a new image containing the ‘Hello World!’ string placed in the map center. Notice that before calling the text.draw() function we configure the map to be centered at specific coordinates (0,0) and zoom level 10 because map parameters such as center and scale are passed as arguments to that text.draw() function. This ensures that the resulting image containing string characters is scaled properly.

      +

      To demonstrate how to use this API, let’s render a simple ‘Hello World!’ text string placed at the map center using default text parameters. The code for this will be:

      +
      // Include the text package.  
+var text = require('users/gena/packages:text');  
+  
+// Configure map (change center and map type).  
+Map.setCenter(0, 0, 10);  
+Map.setOptions('HYBRID');  
+  
+// Draw text string and add to map.  
+var pt = Map.getCenter();  
+var scale = Map.getScale();  
+var image = text.draw('Hello World!', pt, scale);  
+Map.addLayer(image);
      +

      Running the above script will generate a new image containing the ‘Hello World!’ string placed in the map center. Notice that before calling the text.draw() function we configure the map to be centered at specific coordinates (0,0) and zoom level 10 because map parameters such as center and scale are passed as arguments to that text.draw() function. This ensures that the resulting image containing string characters is scaled properly.

      When exporting images containing rendered text strings, it is important to use proper scale to avoid distorted text strings that are difficult to read, depending on the selected font size, as shown in Fig. 6.0.7.

      @@ -594,19 +699,18 @@ Note

      Code Checkpoint F60d. The book’s repository contains a script that shows what your code should look like at this point.

      -

      -

      Fig. 6.0.7 Results of the text.draw call, scaled to 1x: var scale = Map.getScale()1; (left), 2x: var scale = Map.getScale()2; (center), and 0.5x: var scale = Map.getScale()*0.5; (right)

      -

      These artifacts can be avoided to some extent by specifying a larger font size (e.g., 32). However, it is better to render text at the native 1:1 scale to achieve best results. The same applies to the text color and outline: They may need to be adjusted to achieve the best result. Usually, text needs to be rendered using colors that have opposite brightness and colors when compared to the surrounding background. Notice that in the above example, the map was configured to have a dark background (‘HYBRID’) to ensure that the white text (default color) would be visible. Multiple parameters listed in the above API documentation can be used to adjust text rendering. For example, let’s switch font size, font type, text, and outline parameters to render the same string, as below. Replace the existing one-line text.draw call in your script with the following code, and then run it again to see the difference (Fig. F6.0.8):

      -

      var image = text.draw(‘Hello World!’, pt, scale, {
      -   fontSize: 32,
      -   fontType: ‘Consolas’,
      -   textColor: ‘black’,
      -   outlineColor: ‘white’,
      -   outlineWidth: 1,
      -   outlineOpacity: 0.8
      +

      Fig. 6.0.7 Results of the text.draw call, scaled to 1x: var scale = Map.getScale()1; (left), 2x: var scale = Map.getScale()2; (center), and 0.5x: var scale = Map.getScale()*0.5; (right)Fig. 6.0.7 Results of the text.draw call, scaled to 1x: var scale = Map.getScale()1; (left), 2x: var scale = Map.getScale()2; (center), and 0.5x: var scale = Map.getScale()*0.5; (right)Fig. 6.0.7 Results of the text.draw call, scaled to 1x: var scale = Map.getScale()1; (left), 2x: var scale = Map.getScale()2; (center), and 0.5x: var scale = Map.getScale()*0.5; (right)

      +

      These artifacts can be avoided to some extent by specifying a larger font size (e.g., 32). However, it is better to render text at the native 1:1 scale to achieve best results. The same applies to the text color and outline: They may need to be adjusted to achieve the best result. Usually, text needs to be rendered using colors that have opposite brightness and colors when compared to the surrounding background. Notice that in the above example, the map was configured to have a dark background (‘HYBRID’) to ensure that the white text (default color) would be visible. Multiple parameters listed in the above API documentation can be used to adjust text rendering. For example, let’s switch font size, font type, text, and outline parameters to render the same string, as below. Replace the existing one-line text.draw call in your script with the following code, and then run it again to see the difference (Fig. F6.0.8):

      +

      var image = text.draw(‘Hello World!’, pt, scale, {
      +fontSize: 32,
      +fontType: ‘Consolas’,
      +textColor: ‘black’,
      +outlineColor: ‘white’,
      +outlineWidth: 1,
      +outlineOpacity: 0.8
      });

      -

      // Add the text image to the map.
      -Map.addLayer(image);

      +
      // Add the text image to the map.  
+Map.addLayer(image);
      @@ -620,58 +724,69 @@ Note

      Code Checkpoint F60e. The book’s repository contains a script that shows what your code should look like at this point.

      -

      -

      Fig. 6.0.8 Rendering text with adjusted parameters (font type: Consolas, fontSize: 32, textColor: ‘black’, outlineWidth: 1, outlineColor: ‘white’, outlineOpacity: 0.8)

      -

      Of course, non-optional parameters such as pt and scale, as well as the text string, do not have to be hard-coded in the script; instead, they can be acquired by the code using, for example, properties coming from a FeatureCollection. Let’s demonstrate this by showing the cloudiness of Landsat 8 images as text labels rendered in the center of every image. In addition to annotating every image with a cloudiness text string, we will also draw yellow outlines to indicate image boundaries. For convenience, we can also define the code to annotate an image as a function. We will then map that function (as described in Chap. F4.0) over the filtered ImageCollection. The code is as follows:

      -

      var text = require(‘users/gena/packages:text’);

      -

      var geometry = ee.Geometry.Polygon(
      -   [
      -       [
      -           [-109.248, 43.3913],
      -           [-109.248, 33.2689],
      -           [-86.5283, 33.2689],
      -           [-86.5283, 43.3913]
      -       ]
      -   ], null, false);

      +
      +
      +

      +

      Fig. 6.0.8 Rendering text with adjusted parameters (font type: Consolas, fontSize: 32, textColor: ‘black’, outlineWidth: 1, outlineColor: ‘white’, outlineOpacity: 0.8)

      +
      +
      +

      Of course, non-optional parameters such as pt and scale, as well as the text string, do not have to be hard-coded in the script; instead, they can be acquired by the code using, for example, properties coming from a FeatureCollection. Let’s demonstrate this by showing the cloudiness of Landsat 8 images as text labels rendered in the center of every image. In addition to annotating every image with a cloudiness text string, we will also draw yellow outlines to indicate image boundaries. For convenience, we can also define the code to annotate an image as a function. We will then map that function (as described in Chap. F4.0) over the filtered ImageCollection. The code is as follows:

      +

      var text = require(‘users/gena/packages:text’);

      +

      var geometry = ee.Geometry.Polygon(
      +[
      +[
      +[-109.248, 43.3913],
      +[-109.248, 33.2689],
      +[-86.5283, 33.2689],
      +[-86.5283, 43.3913]
      +]
      +], null, false);

      Map.centerObject(geometry, 6);

      -

      function annotate(image) {   // Annotates an image by adding outline border and cloudiness   // Cloudiness is shown as a text string rendered at the image center.   // Add an edge around the image.   var edge = ee.FeatureCollection([image])
      -       .style({
      -           color: ‘cccc00cc’,
      -           fillColor: ‘00000000’       });   // Draw cloudiness as text.   var props = {
      -       textColor: ‘0000aa’,
      -       outlineColor: ‘ffffff’,
      -       outlineWidth: 2,
      -       outlineOpacity: 0.6,
      -       fontSize: 24,
      -       fontType: ‘Consolas’   };   var center = image.geometry().centroid(1);   var str = ee.Number(image.get(‘CLOUD_COVER’)).format(‘%.2f’);   var scale = Map.getScale();   var textCloudiness = text.draw(str, center, scale, props);   // Shift left 25 pixels.   textCloudiness = textCloudiness
      -       .translate(-scale * 25, 0, ‘meters’, ‘EPSG:3857’);   // Merge results.   return ee.ImageCollection([edge, textCloudiness]).mosaic();
      +

      function annotate(image) { // Annotates an image by adding outline border and cloudiness // Cloudiness is shown as a text string rendered at the image center. // Add an edge around the image. var edge = ee.FeatureCollection([image])
      +.style({
      +color: ‘cccc00cc’,
      +fillColor: ‘00000000’ }); // Draw cloudiness as text. var props = {
      +textColor: ‘0000aa’,
      +outlineColor: ‘ffffff’,
      +outlineWidth: 2,
      +outlineOpacity: 0.6,
      +fontSize: 24,
      +fontType: ‘Consolas’ }; var center = image.geometry().centroid(1); var str = ee.Number(image.get(‘CLOUD_COVER’)).format(‘%.2f’); var scale = Map.getScale(); var textCloudiness = text.draw(str, center, scale, props); // Shift left 25 pixels. textCloudiness = textCloudiness
      +.translate(-scale * 25, 0, ‘meters’, ‘EPSG:3857’); // Merge results. return ee.ImageCollection([edge, textCloudiness]).mosaic();
      }

      -

      // Select images.
      -var images = ee.ImageCollection(‘LANDSAT/LC08/C02/T1_RT_TOA’)
      -   .select([5, 4, 2])
      -   .filterBounds(geometry)
      -   .filterDate(‘2018-01-01’, ‘2018-01-7’);

      -

      // dim background.
      -Map.addLayer(ee.Image(1), {
      -   palette: [‘black’]
      -}, ‘black’, true, 0.5);

      -

      // Show images.
      -Map.addLayer(images, {
      -   min: 0.05,
      -   max: 1,
      -   gamma: 1.4}, ‘images’);

      -

      // Show annotations.
      -var labels = images.map(annotate);
      -var labelsLayer = ui.Map.Layer(labels, {}, ‘annotations’);
      -Map.layers().add(labelsLayer);

      -

      The result of defining and mapping this function over the filtered set of images is shown in Fig. F6.0.9. Notice that by adding an outline around the text, we can ensure the text is visible for both dark and light images. Earth Engine requires casting properties to their corresponding value type, which is why we’ve used ee.Number (as described in Chap. F1.0) before generating a formatted string. Also, we have shifted the resulting text image 25 pixels to the left. This was necessary to ensure that the text is positioned properly. In more complex text rendering applications, users may be required to compute the text position in a different way using ee.Geometry calls from the Earth Engine API: for example, by positioning text labels somewhere near the corners.

      -

      -

      Fig. F6.0.9 Annotating Landsat 8 images with image boundaries, border, and text strings indicating cloudiness

      -

      Because we render text labels using the Earth Engine raster API, they are not automatically scaled depending on map zoom size. This may cause unwanted artifacts; To avoid that, the text labels image needs to be updated every time the map zoom changes. To implement this in a script, we can make use of the Map API—in particular, the Map.onChangeZoom event handler. The following code snippet shows how the image containing text annotations can be re-rendered every time the map zoom changes. Add it to the end of your script.

      -

      // re-render (rescale) annotations when map zoom changes.
      -Map.onChangeZoom(function(zoom) {
      -   labelsLayer.setEeObject(images.map(annotate));
      -});

      +
      // Select images.  
+var images = ee.ImageCollection('LANDSAT/LC08/C02/T1_RT_TOA')  
+   .select([5, 4, 2])  
+   .filterBounds(geometry)  
+   .filterDate('2018-01-01', '2018-01-7');  
+  
+// dim background.  
+Map.addLayer(ee.Image(1), {  
+   palette: ['black']  
+}, 'black', true, 0.5);  
+  
+// Show images.  
+Map.addLayer(images, {  
+   min: 0.05,  
+   max: 1,  
+   gamma: 1.4}, 'images');  
+  
+// Show annotations.  
+var labels = images.map(annotate);  
+var labelsLayer = ui.Map.Layer(labels, {}, 'annotations');  
+Map.layers().add(labelsLayer);
      +

      The result of defining and mapping this function over the filtered set of images is shown in Fig. F6.0.9. Notice that by adding an outline around the text, we can ensure the text is visible for both dark and light images. Earth Engine requires casting properties to their corresponding value type, which is why we’ve used ee.Number (as described in Chap. F1.0) before generating a formatted string. Also, we have shifted the resulting text image 25 pixels to the left. This was necessary to ensure that the text is positioned properly. In more complex text rendering applications, users may be required to compute the text position in a different way using ee.Geometry calls from the Earth Engine API: for example, by positioning text labels somewhere near the corners.

      +
      +
      +

      +

      Fig. F6.0.9 Annotating Landsat 8 images with image boundaries, border, and text strings indicating cloudiness

      +
      +
      +

      Because we render text labels using the Earth Engine raster API, they are not automatically scaled depending on map zoom size. This may cause unwanted artifacts; To avoid that, the text labels image needs to be updated every time the map zoom changes. To implement this in a script, we can make use of the Map API—in particular, the Map.onChangeZoom event handler. The following code snippet shows how the image containing text annotations can be re-rendered every time the map zoom changes. Add it to the end of your script.

      +
      // re-render (rescale) annotations when map zoom changes.  
+Map.onChangeZoom(function(zoom) {  
+   labelsLayer.setEeObject(images.map(annotate));  
+});
      @@ -689,8 +804,8 @@ Note

      8.4 Animations

      -

      Visualizing raster images as animations is a useful technique to explore changes in time-dependent datasets, but also, to render short animations to communicate how changing various parameters affects the resulting image—for example, varying thresholds of spectral indices resulting in different binary maps or the changing geometry of vector features.

      -

      Animations are very useful when exploring satellite imagery, as they allow viewers to quickly comprehend dynamics of changes of earth surface or atmospheric properties. Animations can also help to decide what steps should be taken next to designing a robust algorithm to extract useful information from satellite image time series. Earth Engine provides two standard ways to generate animations: as animated GIFs, and as AVI video clips. Animation can also be rendered from a sequence of images exported from Earth Engine, using numerous tools such as ffmpeg or moviepy. However, in many cases it is useful to have a way to quickly explore image collections as animation without requiring extra steps.

      +

      Visualizing raster images as animations is a useful technique to explore changes in time-dependent datasets, but also, to render short animations to communicate how changing various parameters affects the resulting image—for example, varying thresholds of spectral indices resulting in different binary maps or the changing geometry of vector features.

      +

      Animations are very useful when exploring satellite imagery, as they allow viewers to quickly comprehend dynamics of changes of earth surface or atmospheric properties. Animations can also help to decide what steps should be taken next to designing a robust algorithm to extract useful information from satellite image time series. Earth Engine provides two standard ways to generate animations: as animated GIFs, and as AVI video clips. Animation can also be rendered from a sequence of images exported from Earth Engine, using numerous tools such as ffmpeg or moviepy. However, in many cases it is useful to have a way to quickly explore image collections as animation without requiring extra steps.

      In this section, we will generate animations in three different ways:

      1. Generate animated GIF
        2. @@ -698,39 +813,46 @@ Note
      2. Animate image collection interactively using UI controls and map layers

      We will use an image collection showing sea ice as an input dataset to generate animations with visualization parameters from earlier. However, instead of querying a single Sentinel-1 image, let’s generate a filtered image collection with all images intersecting with our area of interest. After importing some packages and palettes and defining a point and rectangle, we’ll build the image collection. Here we will use point geometry to define the location where the image date label will be rendered and the rectangle geometry to indicate the area of interest for the animation. To do this we will build the following logic in a new script. Open a new script and paste the following code into it:

      -

      // Include packages.
      -var palettes = require(‘users/gena/packages:palettes’);
      -var text = require(‘users/gena/packages:text’);

      -

      var point = /* color: #98ff00 */ ee.Geometry.Point([-   106.15944300895228, -74.58262940096245
      -]);

      -

      var rect = /* color: #d63000 */   ee.Geometry.Polygon(
      -       [
      -           [
      -               [-106.19789515738981, -74.56509549360152],
      -               [-106.19789515738981, -74.78071448733921],
      -               [-104.98115931754606, -74.78071448733921],
      -               [-104.98115931754606, -74.56509549360152]
      -           ]
      -       ], null, false);

      -

      // Lookup the ice palette.
      -var palette = palettes.cmocean.Ice[7];

      -

      // Show it in the console.
      -palettes.showPalette(‘Ice’, palette);

      -

      // Center map on geometry.
      -Map.centerObject(point, 9);

      -

      // Select S1 images for the Thwaites glacier.
      -var images = ee.ImageCollection(‘COPERNICUS/S1_GRD’)
      -   .filterBounds(rect)
      -   .filterDate(‘2021-01-01’, ‘2021-03-01’)
      -   .select(‘HH’)   // Make sure we include only images which fully contain the region geometry.   .filter(ee.Filter.isContained({
      -       leftValue: rect,
      -       rightField: ‘.geo’   }))
      -   .sort(‘system:time_start’);

      -

      // Print number of images.
      -print(images.size());

      -

      As you see from the last last lines of the above code, it is frequently useful to print the number of images in an image collection: an example of what’s often known as a “sanity check.”

      -

      Here we have used two custom geometries to configure animations: the green pin named point, used to filter image collection and to position text labels drawn on top of the image, and the blue rectangle rect, used to define a bounding box for the exported animations. To make sure that the point and rectangle geometries are shown under the Geometry Imports in the Code Editor, you need to click on these variables in the code and then select the Convert link.

      -

      Notice that in addition to the bounds and date filter, we have also used a less known isContained filter to ensure that we get only images that fully cover our region. To better understand this filter, you could try commenting out the filter and compare the differences, observing images with empty (masked) pixels in the resulting image collection.

      +
      // Include packages.  
+var palettes = require('users/gena/packages:palettes');  
+var text = require('users/gena/packages:text');  
+  
+var point = /* color: #98ff00 */ ee.Geometry.Point([-   106.15944300895228, -74.58262940096245  
+]);  
+  
+var rect = /* color: #d63000 */   ee.Geometry.Polygon(  
+       [  
+           [  
+               [-106.19789515738981, -74.56509549360152],  
+               [-106.19789515738981, -74.78071448733921],  
+               [-104.98115931754606, -74.78071448733921],  
+               [-104.98115931754606, -74.56509549360152]  
+           ]  
+       ], null, false);  
+  
+// Lookup the ice palette.  
+var palette = palettes.cmocean.Ice[7];  
+  
+// Show it in the console.  
+palettes.showPalette('Ice', palette);  
+  
+// Center map on geometry.  
+Map.centerObject(point, 9);  
+  
+// Select S1 images for the Thwaites glacier.  
+var images = ee.ImageCollection('COPERNICUS/S1_GRD')  
+   .filterBounds(rect)  
+   .filterDate('2021-01-01', '2021-03-01')  
+   .select('HH')   // Make sure we include only images which fully contain the region geometry.   .filter(ee.Filter.isContained({  
+       leftValue: rect,  
+       rightField: '.geo'   }))  
+   .sort('system:time_start');  
+  
+// Print number of images.  
+print(images.size());
      +

      As you see from the last last lines of the above code, it is frequently useful to print the number of images in an image collection: an example of what’s often known as a “sanity check.”

      +

      Here we have used two custom geometries to configure animations: the green pin named point, used to filter image collection and to position text labels drawn on top of the image, and the blue rectangle rect, used to define a bounding box for the exported animations. To make sure that the point and rectangle geometries are shown under the Geometry Imports in the Code Editor, you need to click on these variables in the code and then select the Convert link.

      +

      Notice that in addition to the bounds and date filter, we have also used a less known isContained filter to ensure that we get only images that fully cover our region. To better understand this filter, you could try commenting out the filter and compare the differences, observing images with empty (masked) pixels in the resulting image collection.

      @@ -745,29 +867,35 @@ Note

      Next, to simplify the animation API calls, we will generate a composite RGB image collection out of satellite images and draw the image’s acquisition date as a label on every image, positioned within our region geometry.

      -

      // Render images.
      -var vis = {
      -   palette: palette,
      -   min: -15,
      -   max: 1
      -};

      -

      var scale = Map.getScale();
      -var textProperties = {
      -   outlineColor: ‘000000’,
      -   outlineWidth: 3,
      -   outlineOpacity: 0.6
      -};

      -

      var imagesRgb = images.map(function(i) {   // Use the date as the label.   var label = i.date().format(‘YYYY-MM-dd’);   var labelImage = text.draw(label, point, scale,
      -       textProperties);   return i.visualize(vis)
      -       .blend(labelImage) // Blend label image on top.       .set({
      -           label: label
      -       }); // Keep the text property.
      -});
      -Map.addLayer(imagesRgb.first());
      -Map.addLayer(rect, {color:‘blue’}, ‘rect’, 1, 0.5);

      -

      In addition to printing the size of the ImageCollection, we also often begin by adding a single image to the map from a mapped collection to see that everything works as expected—another example of a sanity check. The resulting map layer will look like F6.0.10.

      -

      -

      Fig. F6.0.10 The results of adding the first layer from the RGB composite image collection showing Sentinel-1 images with a label blended on top at a specified location. The blue geometry is used to define the bounds for the animation to be exported.

      +
      // Render images.  
+var vis = {  
+   palette: palette,  
+   min: -15,  
+   max: 1  
+};  
+  
+var scale = Map.getScale();  
+var textProperties = {  
+   outlineColor: '000000',  
+   outlineWidth: 3,  
+   outlineOpacity: 0.6  
+};  
+  
+var imagesRgb = images.map(function(i) {   // Use the date as the label.   var label = i.date().format('YYYY-MM-dd');   var labelImage = text.draw(label, point, scale,  
+       textProperties);   return i.visualize(vis)  
+       .blend(labelImage) // Blend label image on top.       .set({  
+           label: label  
+       }); // Keep the text property.  
+});  
+Map.addLayer(imagesRgb.first());  
+Map.addLayer(rect, {color:'blue'}, 'rect', 1, 0.5);
      +

      In addition to printing the size of the ImageCollection, we also often begin by adding a single image to the map from a mapped collection to see that everything works as expected—another example of a sanity check. The resulting map layer will look like F6.0.10.

      +
      +
      +

      +

      Fig. F6.0.10 The results of adding the first layer from the RGB composite image collection showing Sentinel-1 images with a label blended on top at a specified location. The blue geometry is used to define the bounds for the animation to be exported.

      +
      +
      @@ -783,58 +911,73 @@ Note

      Animation 1: Animated GIF with ui.Thumbnail

      The quickest way to generate an animation in Earth Engine is to use the animated GIF API and either print it to the Console or print the URL to download the generated GIF. The following code snippet will result in an animated GIF as well as the URL to the animated GIF printed to Console. This is as shown in Fig. F6.0.11:

      -

      // Define GIF visualization parameters.
      -var gifParams = {
      -   region: rect,
      -   dimensions: 600,
      -   crs: ‘EPSG:3857’,
      -   framesPerSecond: 10
      -};

      -

      // Print the GIF URL to the console.
      -print(imagesRgb.getVideoThumbURL(gifParams));

      -

      // Render the GIF animation in the console.
      -print(ui.Thumbnail(imagesRgb, gifParams));

      +
      // Define GIF visualization parameters.  
+var gifParams = {  
+   region: rect,  
+   dimensions: 600,  
+   crs: 'EPSG:3857',  
+   framesPerSecond: 10  
+};  
+  
+// Print the GIF URL to the console.  
+print(imagesRgb.getVideoThumbURL(gifParams));  
+  
+// Render the GIF animation in the console.  
+print(ui.Thumbnail(imagesRgb, gifParams));

      Earth Engine provides multiple options to specify the size of the resulting video. In this example we specify 600 as the size of the maximum dimension. We also specify the number of frames per second for the resulting animated GIF as well as the target projected coordinate system to be used (EPSG:3857 here, which is the projection used in web maps such as Google Maps and the Code Editor background).

      -

      -

      Fig. F6.0.11 Console output after running the animated GIF code snippet, showing the GIF URL and an animation shown directly in the Console

      +
      +
      +

      +

      Fig. F6.0.11 Console output after running the animated GIF code snippet, showing the GIF URL and an animation shown directly in the Console

      +
      +

      Animation 2: Exporting an Animation with Export.video.toDrive

      -

      Animated GIFs can be useful to generate animations quickly. However, they have several limitations. In particular, they are limited to 256 colors, become large for larger animations, and most web players do not provide play controls when playing animated GIFs. To overcome these limitations, Earth Engine provides export of animations as video files in MP4 format. Let’s use the same RGB image collection we have used for the animated GIF to generate a short video. We can ask Earth Engine to export the video to the Google Drive using the following code snippet:

      +

      Animated GIFs can be useful to generate animations quickly. However, they have several limitations. In particular, they are limited to 256 colors, become large for larger animations, and most web players do not provide play controls when playing animated GIFs. To overcome these limitations, Earth Engine provides export of animations as video files in MP4 format. Let’s use the same RGB image collection we have used for the animated GIF to generate a short video. We can ask Earth Engine to export the video to the Google Drive using the following code snippet:

      Export.video.toDrive({
      -   collection: imagesRgb,
      -   description: ‘ice-animation’,
      -   fileNamePrefix: ‘ice-animation’,
      -   framesPerSecond: 10,
      -   dimensions: 600,
      -   region: rect,
      -   crs: ‘EPSG:3857’
      +collection: imagesRgb,
      +description: ‘ice-animation’,
      +fileNamePrefix: ‘ice-animation’,
      +framesPerSecond: 10,
      +dimensions: 600,
      +region: rect,
      +crs: ‘EPSG:3857’
      });

      -

      Here, many arguments to the Export.video.toDrive function resemble the ones we’ve used in the ee.Image.getVideoThumbURL code above. Additional arguments include description and fileNamePrefix, which are required to configure the name of the task and the target file of the video file to be saved to Google Drive. Running the above code will result in a new task created under the Tasks tab in the Code Editor. Starting the export video task (F6.0.12) will result in a video file saved in the Google Drive once completed.

      -

      -

      Fig. F6.0.12 A new export video tasks in the Tasks panel of the Code Editor

      +

      Here, many arguments to the Export.video.toDrive function resemble the ones we’ve used in the ee.Image.getVideoThumbURL code above. Additional arguments include description and fileNamePrefix, which are required to configure the name of the task and the target file of the video file to be saved to Google Drive. Running the above code will result in a new task created under the Tasks tab in the Code Editor. Starting the export video task (F6.0.12) will result in a video file saved in the Google Drive once completed.

      +
      +
      +

      +

      Fig. F6.0.12 A new export video tasks in the Tasks panel of the Code Editor

      +
      +

      Animation 3: The Custom Animation Package

      -

      For the last animation example, we will use the custom package ‘users/gena/packages:animation’, built using the Earth Engine User Interface API. The main difference between this package and the above examples is that it generates an interactive animation by adding Map layers individually to the layer set, and providing UI controls that allow users to play animations or interactively switch between frames. The animate function in that package generates an interactive animation of an ImageCollection, as described below. This function has a number of optional arguments allowing you to configure, for example, the number of frames to be animated, the number of frames to be preloaded, or a few others. The optional parameters to control the function are the following:

      +

      For the last animation example, we will use the custom package ‘users/gena/packages:animation’, built using the Earth Engine User Interface API. The main difference between this package and the above examples is that it generates an interactive animation by adding Map layers individually to the layer set, and providing UI controls that allow users to play animations or interactively switch between frames. The animate function in that package generates an interactive animation of an ImageCollection, as described below. This function has a number of optional arguments allowing you to configure, for example, the number of frames to be animated, the number of frames to be preloaded, or a few others. The optional parameters to control the function are the following:

      • maxFrames: maximum number of frames to show (default: 30)
      • vis: visualization parameters for every frame (default: {})
      • Label: text property of images to show in the animation controls (default: undefined)
        • -
      • width: width of the animation panel (default: ‘600px’)
        • +
      • width: width of the animation panel (default: ‘600px’)
      • compact: show only play control and frame slider (default: false)
      • position: position of the animation panel (default: ‘top-center’)
      • timeStep: time step (ms) used when playing animation (default: 100)
      • preloadCount: number of frames (map layers) to preload (default: all)

      Let’s call this function to add interactive animation controls to the current Map:

      -

      // include the animation package
      -var animation = require(‘users/gena/packages:animation’);

      -

      // show animation controls
      -animation.animate(imagesRgb, {
      - label: ‘label’,
      - maxFrames: 50
      -});

      -

      Before using the interactive animation API, we need to include the corresponding package using require. Here we provide our pre-rendered image collection and two optional parameters (label and maxFrames). The first optional parameter label indicates that every image in our image collection has the ‘label’ text property. The animate function uses this property to name map layers as well as to visualize in the animation UI controls when switching between frames. This can be useful when inspecting image collections. The second optional parameter, maxFrames, indicates that the maximum number of animation frames that we would like to visualize is 50. To prevent the Code Editor from crashing, this parameter should not be too large: it is best to keep it below 100. For a much larger number of frames, it is better to use the Export video or animated GIF API. Running this code snippet will result in the animation control panel added to the map as shown in Fig. F6.0.13.

      +
      // include the animation package  
+var animation = require('users/gena/packages:animation');  
+  
+// show animation controls  
+animation.animate(imagesRgb, {  
+ label: 'label',  
+ maxFrames: 50  
+});
      +

      Before using the interactive animation API, we need to include the corresponding package using require. Here we provide our pre-rendered image collection and two optional parameters (label and maxFrames). The first optional parameter label indicates that every image in our image collection has the ‘label’ text property. The animate function uses this property to name map layers as well as to visualize in the animation UI controls when switching between frames. This can be useful when inspecting image collections. The second optional parameter, maxFrames, indicates that the maximum number of animation frames that we would like to visualize is 50. To prevent the Code Editor from crashing, this parameter should not be too large: it is best to keep it below 100. For a much larger number of frames, it is better to use the Export video or animated GIF API. Running this code snippet will result in the animation control panel added to the map as shown in Fig. F6.0.13.

      It is important to note that the animation API uses asynchronous UI calls to make sure that the Code Editor does not hang when running the script. The drawback of this is that for complex image collections, a large amount of processing is required. Hence, it may take some time to process all images and to visualize the interactive animation panel. The same is true for map layer names: they are updated once the animation panel is visualized. Also, map layers used to visualize individual images in the provided image collection may require some time to be rendered.

      -

      -

      Fig. F6.0.13 Interactive animation controls when using custom animation API

      +
      +
      +

      +

      Fig. F6.0.13 Interactive animation controls when using custom animation API

      +
      +

      The main advantage of the interactive animation API is that it provides a way to explore image collections at frame-by-frame basis, which can greatly improve our visual understanding of the changes captured in sets of images.

      @@ -854,71 +997,72 @@ Note

      8.5 Terrain Visualization

      This section introduces several raster visualization techniques useful to visualize terrain data such as:

        -
      • Basic hillshading and parameters (light azimuth, elevation)
        • -
      • Combining elevation data and colors using HSV transform (Wikipedia: HSL and HSV)
        • +
      • Basic hillshading and parameters (light azimuth, elevation)
        • +
      • Combining elevation data and colors using HSV transform (Wikipedia: HSL and HSV)
      • Adding shadows
      -

      One special type of raster data is data that represents height. Elevation data can include topography, bathymetry, but also other forms of height, such as sea surface height can be presented as a terrain.

      +

      One special type of raster data is data that represents height. Elevation data can include topography, bathymetry, but also other forms of height, such as sea surface height can be presented as a terrain.

      Height is often visualized using the concept of directional light with a technique called hillshading. Because height is such a common feature in our environment, we also have an expectancy of how height is visualized. If height is visualized using a simple grayscale colormap, it looks very unnatural (Fig. F6.0.14, top left). By using hillshading, data immediately looks more natural (Fig. F6.0.14, top middle).

      -

      We can further improve the visualization by including shadows (Fig. F6.0.14, top right).  A final step is to replace the simple grayscale colormap with a perceptual uniform topographic colormap and mix this with the hillshading and shadows (Fig. F6.0.14, bottom). This section explains how to apply these techniques.

      +

      We can further improve the visualization by including shadows (Fig. F6.0.14, top right). A final step is to replace the simple grayscale colormap with a perceptual uniform topographic colormap and mix this with the hillshading and shadows (Fig. F6.0.14, bottom). This section explains how to apply these techniques.

      We’ll focus on elevation data stored in raster form. Elevation data is not always stored in raster formats. Other data formats include Triangulated Irregular Network (TIN), which allows storing information at varying resolutions and as 3D objects. This format allows one to have overlapping geometries, such as bridges with a road below it. In raster-based digital elevation models, in contrast, there can only be one height recorded for each pixel.

      Let’s start by loading data from a digital elevation model. This loads a topographic dataset from the Netherlands (Algemeen Hoogtebestand Nederland). It is a Digital Surface Model, based on airborne LIDAR measurements regridded to 0.5 m resolution. Enter the following code in a new script.

      -

      var dem = ee.Image(‘AHN/AHN2_05M_RUW’);

      -

      We can visualize this dataset using a sequential gradient colormap from black to white. This results in Fig. F6.0.14. One can infer which areas are lower and which are higher, but the visualization does not quite “feel” like a terrain.

      -

      // Change map style to HYBRID and center map on the Netherlands
      -Map.setOptions(‘HYBRID’);
      -Map.setCenter(4.4082, 52.1775, 18);

      -

      // Visualize DEM using black-white color palette
      -var palette = [‘black’, ‘white’];
      -var demRGB = dem.visualize({
      -   min: -5,
      -   max: 5,
      -   palette: palette
      -});
      -Map.addLayer(demRGB, {},‘DEM’);

      -

      An important step to visualize terrain is to add shadows created by a distant point source of light. This is referred to as hillshading or a shaded relief map. This type of map became popular in the 1940s through the work of Edward Imhoff, who also used grayscale colormaps (Imhoff, 2015). Here we’ll use the ‘gena/packages:utils’ library to combine the colormap image with the shadows. That Earth Engine package implements a hillshadeRGB function to simplify rendering of images enhanced with hillshading and shadow effects. One important argument this function takes is the light azimuth—an angle from the image plane upward to the light source (the Sun). This should always be set to the top left to avoid bistable perception artifacts, in which the DEM can be misperceived as inverted.

      -

      var utils = require(‘users/gena/packages:utils’);

      -

      var weight =   0.4; // Weight of Hillshade vs RGB (0 - flat, 1 - hillshaded).
      -var exaggeration = 5; // Vertical exaggeration.
      -var azimuth = 315; // Sun azimuth.
      -var zenith = 20; // Sun elevation.
      -var brightness = -0.05; // 0 - default.
      -var contrast = 0.05; // 0 - default.
      -var saturation = 0.8; // 1 - default.
      -var castShadows = false;

      -

      var rgb = utils.hillshadeRGB(
      -   demRGB, dem, weight, exaggeration, azimuth, zenith,
      -   contrast, brightness, saturation, castShadows);

      +

      var dem = ee.Image(‘AHN/AHN2_05M_RUW’);

      +

      We can visualize this dataset using a sequential gradient colormap from black to white. This results in Fig. F6.0.14. One can infer which areas are lower and which are higher, but the visualization does not quite “feel” like a terrain.

      +
      // Change map style to HYBRID and center map on the Netherlands  
+Map.setOptions('HYBRID');  
+Map.setCenter(4.4082, 52.1775, 18);  
+  
+// Visualize DEM using black-white color palette  
+var palette = ['black', 'white'];  
+var demRGB = dem.visualize({  
+   min: -5,  
+   max: 5,  
+   palette: palette  
+});  
+Map.addLayer(demRGB, {},'DEM');
      +

      An important step to visualize terrain is to add shadows created by a distant point source of light. This is referred to as hillshading or a shaded relief map. This type of map became popular in the 1940s through the work of Edward Imhoff, who also used grayscale colormaps (Imhoff, 2015). Here we’ll use the ‘gena/packages:utils’ library to combine the colormap image with the shadows. That Earth Engine package implements a hillshadeRGB function to simplify rendering of images enhanced with hillshading and shadow effects. One important argument this function takes is the light azimuth—an angle from the image plane upward to the light source (the Sun). This should always be set to the top left to avoid bistable perception artifacts, in which the DEM can be misperceived as inverted.

      +

      var utils = require(‘users/gena/packages:utils’);

      +

      var weight = 0.4; // Weight of Hillshade vs RGB (0 - flat, 1 - hillshaded).
      +var exaggeration = 5; // Vertical exaggeration.
      +var azimuth = 315; // Sun azimuth.
      +var zenith = 20; // Sun elevation.
      +var brightness = -0.05; // 0 - default.
      +var contrast = 0.05; // 0 - default.
      +var saturation = 0.8; // 1 - default.
      +var castShadows = false;

      +

      var rgb = utils.hillshadeRGB(
      +demRGB, dem, weight, exaggeration, azimuth, zenith,
      +contrast, brightness, saturation, castShadows);

      Map.addLayer(rgb, {}, ‘DEM (no shadows)’);

      -

      Standard hillshading only determines per pixel if it will be directed to the light or not. One can also project shadows on the map. That is done using the ee.Algorithms.HillShadow algorithm. Here we’ll turn on castShadows in the hillshadeRGB function. This results in a more realistic map, as can be seen in Figure F6.0.14.

      -

      var castShadows = true;

      -

      var rgb = utils.hillshadeRGB(
      -   demRGB, dem, weight, exaggeration, azimuth, zenith,
      -   contrast, brightness, saturation, castShadows);

      +

      Standard hillshading only determines per pixel if it will be directed to the light or not. One can also project shadows on the map. That is done using the ee.Algorithms.HillShadow algorithm. Here we’ll turn on castShadows in the hillshadeRGB function. This results in a more realistic map, as can be seen in Figure F6.0.14.

      +

      var castShadows = true;

      +

      var rgb = utils.hillshadeRGB(
      +demRGB, dem, weight, exaggeration, azimuth, zenith,
      +contrast, brightness, saturation, castShadows);

      Map.addLayer(rgb, {}, ‘DEM (with shadows)’);

      -

      The final step is to add a topographic colormap. To visualize topographic information, one often uses special topographic colormaps. Here we’ll use the oleron colormap from crameri. The colors get mixed with the shadows using the hillshadeRGB function. As you can see in Fig. F6.0.14, this gives a nice overview of the terrain. The area colored in blue is located below sea level.

      -

      var palettes = require(‘users/gena/packages:palettes’);
      -var palette = palettes.crameri.oleron[50];

      -

      var demRGB = dem.visualize({
      -   min: -5,
      -   max: 5,
      -   palette: palette
      +

      The final step is to add a topographic colormap. To visualize topographic information, one often uses special topographic colormaps. Here we’ll use the oleron colormap from crameri. The colors get mixed with the shadows using the hillshadeRGB function. As you can see in Fig. F6.0.14, this gives a nice overview of the terrain. The area colored in blue is located below sea level.

      +

      var palettes = require(‘users/gena/packages:palettes’);
      +var palette = palettes.crameri.oleron[50];

      +

      var demRGB = dem.visualize({
      +min: -5,
      +max: 5,
      +palette: palette
      });

      -

      var castShadows = true;

      -

      var rgb = utils.hillshadeRGB(
      -   demRGB, dem, weight, exaggeration, azimuth, zenith,
      -   contrast, brightness, saturation, castShadows);

      +

      var castShadows = true;

      +

      var rgb = utils.hillshadeRGB(
      +demRGB, dem, weight, exaggeration, azimuth, zenith,
      +contrast, brightness, saturation, castShadows);

      Map.addLayer(rgb, {}, ‘DEM colormap’);

      -

      Steps to further improve a terrain visualization include using light sources from multiple directions. This allows the user to render terrain to appear more natural. In the real world light is often scattered by clouds and other reflections.

      -

      One can also use lights to emphasize certain regions. To use even more advanced lighting techniques one can use a raytracing engine, such as the R rayshader library, as discussed earlier in this chapter. The raytracing engine in the Blender 3D program is also capable of producing stunning terrain visualizations using physical-based rendering, mist, environment lights, and camera effects such as depth of field.

      +

      Steps to further improve a terrain visualization include using light sources from multiple directions. This allows the user to render terrain to appear more natural. In the real world light is often scattered by clouds and other reflections.

      +

      One can also use lights to emphasize certain regions. To use even more advanced lighting techniques one can use a raytracing engine, such as the R rayshader library, as discussed earlier in this chapter. The raytracing engine in the Blender 3D program is also capable of producing stunning terrain visualizations using physical-based rendering, mist, environment lights, and camera effects such as depth of field.

      -

      Figure F6.0.14 Hillshading with shadows

      +

      Figure F6.0.14 Hillshading with shadows

      Steps in visualizing a topographic dataset:

      1. Top left, topography with grayscale colormap
      2. Top middle, topography with grayscale colormap and hillshading
        3. -
      3. Top right, topography with grayscale colormap, hillshading, and shadows
        4. +
      4. Top right, topography with grayscale colormap, hillshading, and shadows
      5. Bottom, topography with topographic colormap, hillshading, and shadows
      @@ -931,16 +1075,16 @@ Note
      -

      Code Checkpoint F60j. The book’s repository contains a script that shows what your code should look like at this point.

      +

      Code Checkpoint F60j. The book’s repository contains a script that shows what your code should look like at this point.

      Synthesis

      To synthesize what you have learned in this chapter, you can do the following assignments.

      -

      Assignment 1. Experiment with different color palettes from the palettes library. Try combining palettes with image opacity (using ee.Image.updateMask call) to visualize different physical features (for example, hot or cold areas using temperature and elevation).

      -

      Assignment 2. Render multiple text annotations when generating animations using image collection. For example, show other image properties in addition to date or image statistics generated using regional reducers for every image.

      -

      Assignment 3. In addition to text annotations, try blending geometry elements (lines, polygons) to highlight specific areas of rendered images.

      +

      Assignment 1. Experiment with different color palettes from the palettes library. Try combining palettes with image opacity (using ee.Image.updateMask call) to visualize different physical features (for example, hot or cold areas using temperature and elevation).

      +

      Assignment 2. Render multiple text annotations when generating animations using image collection. For example, show other image properties in addition to date or image statistics generated using regional reducers for every image.

      +

      Assignment 3. In addition to text annotations, try blending geometry elements (lines, polygons) to highlight specific areas of rendered images.

      Conclusion

      @@ -960,7 +1104,7 @@ Note
      -

      9 Collaborating in Earth Engine with Scripts and Assets

      +

      9 Collaborating in Earth Engine with Scripts and Assets

      @@ -990,7 +1134,7 @@ Chapter Information
    • Sharing a repository with others.
    • Seeing who made changes to a script and what changes were made.
    • Reverting to a previous version of a script.
      • -
    • Using the require function to load modules.
      • +
    • Using the require function to load modules.
    • Creating a script to share as a module.
    @@ -1005,214 +1149,267 @@ Chapter Information

    Introduction

    Many people find themselves needing to share a script when they encounter a problem; they wish to share the script with someone else so they can ask a question. When this occurs, sharing a link to the script often suffices. The other person can then make comments or changes before sending a new link to the modified script.

    -

    If you have included any assets from the Asset Manager in your script, you will also need to share these assets in order for your script to work for your colleague. The same goes for sharing assets to be displayed in an app.

    +

    If you have included any assets from the Asset Manager in your script, you will also need to share these assets in order for your script to work for your colleague. The same goes for sharing assets to be displayed in an app.

    Another common situation involves collaborating with others on a project. They may have some scripts they have written that they want to reuse or modify for the new project. Alternatively, several people might want to work on the same script together. For this situation, sharing a repository would be the best way forward; team members will be able to see who made what changes to a script and even revert to a previous version.

    -

    If you or your group members find yourselves repeatedly reusing certain functions for visualization or for part of your analysis, you could use the require module to call that function instead of having to copy and paste it into a new script each time. You could even make this function or module available to others to use via require.

    +

    If you or your group members find yourselves repeatedly reusing certain functions for visualization or for part of your analysis, you could use the require module to call that function instead of having to copy and paste it into a new script each time. You could even make this function or module available to others to use via require.

    Let’s get started. For this lab, you will need to work in small groups or pairs.

    9.2 Sharing Assets from Your Asset Manager

    -

    When you clicked the Get Link button earlier, you may have noticed a note in the popup reading: “To give others access to assets in the code snapshot, you may need to share them.” If your script uses an asset that you have uploaded into your Asset Manager, you will need to share that asset as well. If not, an error message will appear when the person you shared the script with tries to run it.

    +

    When you clicked the Get Link button earlier, you may have noticed a note in the popup reading: “To give others access to assets in the code snapshot, you may need to share them.” If your script uses an asset that you have uploaded into your Asset Manager, you will need to share that asset as well. If not, an error message will appear when the person you shared the script with tries to run it.

    Before sharing an asset, think about whether you have permission to share it. Is this some data that is owned by you, or did you get it from somewhere else? Do you need permission to share this asset? Make sure you have the permission to share an asset before doing so.

    -

    Now, let’s practice sharing assets. First, navigate to your Asset Manager by clicking on the Assets tab in the left panel. If you already have some assets uploaded, pick one that you have permission to share. If not, upload one to your Asset Manager. If you don’t have a shapefile or raster to upload, you can upload a small text file. Consult the Earth Engine documentation for how to do this; it will take only a few steps.

    -

    Hover your cursor over that asset in your Asset Manager. The asset gets highlighted in gray and three buttons appear to the right of the asset. Click on the first button from the left (outlined in red in Fig. F6.1.1). This icon means “share.”

    -

    -

    Fig. F6.1.1 Three assets in the Asset Manager

    -

    After you click the share button, a Share Image popup will appear (Fig. F6.1.2). This popup contains information about the path of the asset and the email address of the owner. The owner of the asset can decide who can view and edit the asset.

    -

    Click on the dropdown menu outlined in red in Fig. F6.1.2. You will see two options for permissions: Reader and Writer. A Reader can view the asset, while a Writer can both view and make changes to it. For example, a Writer could add a new image to an ImageCollection. A Writer can also add other people to view or edit the asset, and a Writer can delete the asset. When in doubt, give someone the Reader role rather than the Writer role.

    -

    -

    Fig. F6.1.2 The Share Image popup window

    -

    To share an asset with someone, you can type their email address into the Email or domain text field, choose Reader or Writer in the dropdown menu, and then click on Add Access. You can also share an asset with everyone with a certain email domain, which is useful if you want to share an asset with everyone in your organization, for instance.

    -

    If you want to share reading access publicly, then check the box that says Anyone can read. Note that you still need to share the link to the asset in order for others to access it. The only exceptions to this are when you are using the asset in a script and sharing that script using the Get Link button or when you share the asset with an Earth Engine app. To do the latter, use the Select an app dropdown menu (outlined in orange in Fig. F6.1.2) and click Add App Access.

    +

    Now, let’s practice sharing assets. First, navigate to your Asset Manager by clicking on the Assets tab in the left panel. If you already have some assets uploaded, pick one that you have permission to share. If not, upload one to your Asset Manager. If you don’t have a shapefile or raster to upload, you can upload a small text file. Consult the Earth Engine documentation for how to do this; it will take only a few steps.

    +

    Hover your cursor over that asset in your Asset Manager. The asset gets highlighted in gray and three buttons appear to the right of the asset. Click on the first button from the left (outlined in red in Fig. F6.1.1). This icon means “share.”

    +
    +
    +

    +

    Fig. F6.1.1 Three assets in the Asset Manager

    +
    +
    +

    After you click the share button, a Share Image popup will appear (Fig. F6.1.2). This popup contains information about the path of the asset and the email address of the owner. The owner of the asset can decide who can view and edit the asset.

    +

    Click on the dropdown menu outlined in red in Fig. F6.1.2. You will see two options for permissions: Reader and Writer. A Reader can view the asset, while a Writer can both view and make changes to it. For example, a Writer could add a new image to an ImageCollection. A Writer can also add other people to view or edit the asset, and a Writer can delete the asset. When in doubt, give someone the Reader role rather than the Writer role.

    +
    +
    +

    +

    Fig. F6.1.2 The Share Image popup window

    +
    +
    +

    To share an asset with someone, you can type their email address into the Email or domain text field, choose Reader or Writer in the dropdown menu, and then click on Add Access. You can also share an asset with everyone with a certain email domain, which is useful if you want to share an asset with everyone in your organization, for instance.

    +

    If you want to share reading access publicly, then check the box that says Anyone can read. Note that you still need to share the link to the asset in order for others to access it. The only exceptions to this are when you are using the asset in a script and sharing that script using the Get Link button or when you share the asset with an Earth Engine app. To do the latter, use the Select an app dropdown menu (outlined in orange in Fig. F6.1.2) and click Add App Access.

    Question 3. Share an asset with a group member and give them reader access. Send them the link to that asset. You will also receive a link from someone else in your group. Open that link. What can you do with that asset? What do you need to do to import it into a script?

    -

    Answer: You can view details about the asset and import it for use in a script in the Code Editor. To import the asset, click on the blue Import button.

    +

    Answer: You can view details about the asset and import it for use in a script in the Code Editor. To import the asset, click on the blue Import button.

    Question 4. Share an asset with a group member and give them writer access. Send them the link to that asset. You will also receive a link from someone else in your group. Open that link. What can you do with that asset? Try sharing the asset with a different group member.

    -

    Answer: You can view details about the asset and import it for use in a script in the Code Editor. You can also share the asset with others and delete the asset.

    +

    Answer: You can view details about the asset and import it for use in a script in the Code Editor. You can also share the asset with others and delete the asset.

    9.3 Working with Shared Repositories

    Now that you know how to share assets and scripts, let’s move on to sharing repositories. In this section, you will learn about different types of repositories and how to add a repository that someone else shared with you. You will also learn how to view previous versions of a script and how to revert back to an earlier version.

    -

    Earlier, we learned how to share a script using the Get Link button. This link shares a code snapshot from a script. This snapshot does not reflect any changes made to the script after the time the link was shared. If you want to share a script that updates to reflect the most current version when it is opened, you need to share a repository with that script instead.

    -

    If you look under the Scripts tab of the leftmost panel in the Code Editor, you will see that the first three categories are labeled Owner, Reader, and Writer.

    +

    Earlier, we learned how to share a script using the Get Link button. This link shares a code snapshot from a script. This snapshot does not reflect any changes made to the script after the time the link was shared. If you want to share a script that updates to reflect the most current version when it is opened, you need to share a repository with that script instead.

    +

    If you look under the Scripts tab of the leftmost panel in the Code Editor, you will see that the first three categories are labeled Owner, Reader, and Writer.

      -
    • Repositories categorized under Owner are created and owned by you. No one else has access to view or make changes to them until you share these repositories.
      • -
    • Repositories categorized under Reader are repositories to which you have reader access. You can view the scripts but not make any changes to them. If you want to make any changes, you will need to save the script as a new file in a repository that you own.
      • -
    • Repositories categorized under Writer are repositories to which you have writer access. This means you can view and make changes to the scripts.
      • +
    • Repositories categorized under Owner are created and owned by you. No one else has access to view or make changes to them until you share these repositories.
      • +
    • Repositories categorized under Reader are repositories to which you have reader access. You can view the scripts but not make any changes to them. If you want to make any changes, you will need to save the script as a new file in a repository that you own.
      • +
    • Repositories categorized under Writer are repositories to which you have writer access. This means you can view and make changes to the scripts.
    -

    Let’s practice creating and sharing repositories. We will start by making a new repository. Click on the red New button located in the left panel. Select Repository from the dropdown menu. A New repository popup window will open (Fig. F6.1.3).

    -

    -

    Fig. F6.1.3 The New repository popup window

    -

    In the popup window’s text field, type a name for your new repository, such as “ForSharing1,” then click on the blue Create button. You will see the new repository appear under the Owner category in the Scripts tab (Fig. F6.1.4).

    -

    Now, share this new repository with your group members: Hover your cursor over the repository you want to share. The repository gets highlighted in gray, and three buttons appear. Click on the Gear icon (outlined in red in Fig. F6.1.4).

    -

    -

    Fig. F6.1.4 Three repositories under the Owner category

    -

    A Share Repo popup window appears (Fig. F6.1.5) which is very similar to the Share Image popup window we saw in Fig. F6.1.2. The method for sharing a repository with a specific user or the general public is the same as for sharing assets.

    -

    Type the email address of a group member in the Email or domain text field and give this person a writer role by selecting Writer in the dropdown menu, then click on Add Access. 

    -

    -

    Fig. F6.1.5. The Share Repo popup window

    +

    Let’s practice creating and sharing repositories. We will start by making a new repository. Click on the red New button located in the left panel. Select Repository from the dropdown menu. A New repository popup window will open (Fig. F6.1.3).

    +
    +
    +

    +

    Fig. F6.1.3 The New repository popup window

    +
    +
    +

    In the popup window’s text field, type a name for your new repository, such as “ForSharing1,” then click on the blue Create button. You will see the new repository appear under the Owner category in the Scripts tab (Fig. F6.1.4).

    +

    Now, share this new repository with your group members: Hover your cursor over the repository you want to share. The repository gets highlighted in gray, and three buttons appear. Click on the Gear icon (outlined in red in Fig. F6.1.4).

    +
    +
    +

    +

    Fig. F6.1.4 Three repositories under the Owner category

    +
    +
    +

    A Share Repo popup window appears (Fig. F6.1.5) which is very similar to the Share Image popup window we saw in Fig. F6.1.2. The method for sharing a repository with a specific user or the general public is the same as for sharing assets.

    +

    Type the email address of a group member in the Email or domain text field and give this person a writer role by selecting Writer in the dropdown menu, then click on Add Access.

    +
    +
    +

    +

    Fig. F6.1.5. The Share Repo popup window

    +
    +

    Your group member should receive an email inviting them to edit the repository. Check your email inbox for the repository that your group member has shared with you. When you open that email, you will see content similar to what is shown in Fig. F6.1.6.

    -

    -

    Fig. F6.1.6 The “Invitation to edit” email

    -

    Now, click on the blue button that says Add [repository path] to your Earth Engine Code Editor. You will find the new repository added to the Writer category in your Scripts tab. The repository path will contain the username of your group member, such as users/username/sharing.

    -

    Now, let’s add a script to the empty repository. Click on the red New button in the Scripts tab and select File from the dropdown menu. A Create file popup will appear, as shown in Fig. F6.1.7. Click on the gray arrow beside the default path to open a dropdown menu that will allow you to choose the path of the repository that your group member shared with you. Type a new File Name in the text field, such as “exercise,” then click on the blue OK button to create the file.

    -

    -

    Fig. F6.1.7 The Create file popup window

    -

    A new file should now appear in the shared repository in the Writer category. If you don’t see it, click on the Refresh icon, which is to the right of the red New button in the Scripts tab.

    +
    +
    +

    +

    Fig. F6.1.6 The “Invitation to edit” email

    +
    +
    +

    Now, click on the blue button that says Add [repository path] to your Earth Engine Code Editor. You will find the new repository added to the Writer category in your Scripts tab. The repository path will contain the username of your group member, such as users/username/sharing.

    +

    Now, let’s add a script to the empty repository. Click on the red New button in the Scripts tab and select File from the dropdown menu. A Create file popup will appear, as shown in Fig. F6.1.7. Click on the gray arrow beside the default path to open a dropdown menu that will allow you to choose the path of the repository that your group member shared with you. Type a new File Name in the text field, such as “exercise,” then click on the blue OK button to create the file.

    +
    +
    +

    +

    Fig. F6.1.7 The Create file popup window

    +
    +
    +

    A new file should now appear in the shared repository in the Writer category. If you don’t see it, click on the Refresh icon, which is to the right of the red New button in the Scripts tab.

    Double-click on the new script in the shared repository to open it. Then, copy and paste the following code to your Code Editor.

    print(‘The owner of this repository is GroupMemberName.’);

    -

    Replace GroupMemberName with the name of your group member, then click Save to save the script in the shared repository, which is under the Writer category.

    -

    Now, navigate to the repository under Owner which you shared with your group member. Open the new script which they just created by double-clicking it.

    -

    Add the following code below the line of code that you pasted earlier.

    +

    Replace GroupMemberName with the name of your group member, then click Save to save the script in the shared repository, which is under the Writer category.

    +

    Now, navigate to the repository under Owner which you shared with your group member. Open the new script which they just created by double-clicking it.

    +

    Add the following code below the line of code that you pasted earlier.

    print(‘This script is shared with MyName.’);

    -

    Replace MyName with your name, then save the script.

    -

    Next, we will compare changes made to the script. Click on the Versions icon (outlined in red in Fig. F6.1.8).

    -

    -

    Fig. F6.1.8 Changes made and previous versions of the script

    -

    A popup window will appear, titled Revision history, followed by the path of the script (Fig. F6.1.9). There are three columns of information below the title.

    +

    Replace MyName with your name, then save the script.

    +

    Next, we will compare changes made to the script. Click on the Versions icon (outlined in red in Fig. F6.1.8).

    +
    +
    +

    +

    Fig. F6.1.8 Changes made and previous versions of the script

    +
    +
    +

    A popup window will appear, titled Revision history, followed by the path of the script (Fig. F6.1.9). There are three columns of information below the title.

      -
    • The left column contains the dates on which changes have  been made.
      • +
    • The left column contains the dates on which changes have been made.
    • The middle column contains the usernames of the people who made changes.
      • -
    • The right column contains information about what changes were made.  
      • +
    • The right column contains information about what changes were made.

    The most recent version of the script is shown in the first row, while previous versions are listed in subsequent rows. (More advanced users may notice that this is actually a Git repository.)

    -

    -

    Fig. F6.1.9 The Revision history popup window

    -

    If you hover your cursor over a row, the row will be highlighted in gray and a button labeled Compare will appear. Clicking on this button allows you to compare differences between the current version of the script and a previous version in a Version comparison popup window (Fig. F6.1.10).

    -

    -

    Fig. F6.1.10 The Version comparison popup window

    -

    In the Version comparison popup, you will see text highlighted in two different colors. Text highlighted in red shows code that was present in the older version but is absent in the current version (the “latest commit”). Text highlighted in green shows code that is present in the current version but that was absent in the older version. Generally speaking, text highlighted in red has been removed in the current version and text highlighted in green has been added to the current version. Text that is not highlighted shows code that is present in both versions.

    -

    Question 5. What text, if any, is highlighted in red when you click on Compare in your “exercise” script?

    -

    Answer: No text is highlighted in red, because none was removed between the previous and current versions of the script.

    -

    Question 6. What text, if any, is highlighted in green when you click on Compare in your “exercise” script?

    -

    Answer: print(‘This script is shared with MyName.’);

    -

    Question 7. What happens when you click on the blue Revert button?

    -

    Answer: The script reverts to the previous version, in which the only line of code is

    +
    +
    +

    +

    Fig. F6.1.9 The Revision history popup window

    +
    +
    +

    If you hover your cursor over a row, the row will be highlighted in gray and a button labeled Compare will appear. Clicking on this button allows you to compare differences between the current version of the script and a previous version in a Version comparison popup window (Fig. F6.1.10).

    +
    +
    +

    +

    Fig. F6.1.10 The Version comparison popup window

    +
    +
    +

    In the Version comparison popup, you will see text highlighted in two different colors. Text highlighted in red shows code that was present in the older version but is absent in the current version (the “latest commit”). Text highlighted in green shows code that is present in the current version but that was absent in the older version. Generally speaking, text highlighted in red has been removed in the current version and text highlighted in green has been added to the current version. Text that is not highlighted shows code that is present in both versions.

    +

    Question 5. What text, if any, is highlighted in red when you click on Compare in your “exercise” script?

    +

    Answer: No text is highlighted in red, because none was removed between the previous and current versions of the script.

    +

    Question 6. What text, if any, is highlighted in green when you click on Compare in your “exercise” script?

    +

    Answer: print(‘This script is shared with MyName.’);

    +

    Question 7. What happens when you click on the blue Revert button?

    +

    Answer: The script reverts to the previous version, in which the only line of code is

    print(‘The owner of this repository is GroupMemberName.’);

    9.4 Using the Require Function to Load a Module

    -

    In earlier chapters, you may have noticed that the require function allows you to reuse code that has already been written without having to copy and paste it into your current script. For example, you might have written a function for cloud masking that you would like to use in multiple scripts. Saving this function as a module enables you to share the code across your own scripts and with other people. Or you might discover a new module with capabilities you need written by other authors. This section will show you how to use the require function to create and share your own module or to load a module that someone else has shared.

    -

    The module we will use is ee-palettes, which enables users to visualize raster data using common specialized color palettes (Donchyts et al. 2019). (If you would like to learn more about using these color palettes, the ee-palettes module is described and illustrated in detail in Chap. F6.0.) The first step is to go to this link to accept access to the repository as a reader: https://code.earthengine.google.com/?accept_repo=users/gena/packages 

    -

    Now, if you navigate to your Reader directory in the Code Editor, you should see a new repository called ‘users/gena/packages’ listed. Look for a script called ‘palettes’ and click on it to load it in your Code Editor.

    -

    If you scroll down, you will see that the script contains a nested series of dictionaries with lists of hexadecimal color specifications (as described in Chap. F2.1) that describe a color palette, as shown in the code block below. For example, the color palette named “Algae” stored in the cmocean variable consists of seven colors, ranging from dark green to light green (Fig. F6.1.11).

    -

    exports.cmocean = {
    -   Thermal: {       7: [‘042333’, ‘2c3395’, ‘744992’, ‘b15f82’, ‘eb7958’,           ‘fbb43d’, ‘e8fa5b’       ]
    -   },
    -   Haline: {       7: [‘2a186c’, ‘14439c’, ‘206e8b’, ‘3c9387’, ‘5ab978’,           ‘aad85c’, ‘fdef9a’       ]
    -   },
    -   Solar: {       7: [‘331418’, ‘682325’, ‘973b1c’, ‘b66413’, ‘cb921a’,           ‘dac62f’, ‘e1fd4b’       ]
    -   },
    -   Ice: {       7: [‘040613’, ‘292851’, ‘3f4b96’, ‘427bb7’, ‘61a8c7’,           ‘9cd4da’, ‘eafdfd’       ]
    -   },
    -   Gray: {       7: [‘000000’, ‘232323’, ‘4a4a49’, ‘727171’, ‘9b9a9a’,           ‘cacac9’, ‘fffffd’       ]
    -   },
    -   Oxy: {       7: [‘400505’, ‘850a0b’, ‘6f6f6e’, ‘9b9a9a’, ‘cbcac9’,           ‘ebf34b’, ‘ddaf19’       ]
    -   },
    -   Deep: {       7: [‘fdfecc’, ‘a5dfa7’, ‘5dbaa4’, ‘488e9e’, ‘3e6495’,           ‘3f396c’, ‘281a2c’       ]
    -   },
    -   Dense: {       7: [‘e6f1f1’, ‘a2cee2’, ‘76a4e5’, ‘7871d5’, ‘7642a5’,           ‘621d62’, ‘360e24’       ]
    -   },
    -   Algae: {       7: [‘d7f9d0’, ‘a2d595’, ‘64b463’, ‘129450’, ‘126e45’,           ‘1a482f’, ‘122414’       ]
    -   },
    -   …
    +

    In earlier chapters, you may have noticed that the require function allows you to reuse code that has already been written without having to copy and paste it into your current script. For example, you might have written a function for cloud masking that you would like to use in multiple scripts. Saving this function as a module enables you to share the code across your own scripts and with other people. Or you might discover a new module with capabilities you need written by other authors. This section will show you how to use the require function to create and share your own module or to load a module that someone else has shared.

    +

    The module we will use is ee-palettes, which enables users to visualize raster data using common specialized color palettes (Donchyts et al. 2019). (If you would like to learn more about using these color palettes, the ee-palettes module is described and illustrated in detail in Chap. F6.0.) The first step is to go to this link to accept access to the repository as a reader: https://code.earthengine.google.com/?accept_repo=users/gena/packages

    +

    Now, if you navigate to your Reader directory in the Code Editor, you should see a new repository called ‘users/gena/packages’ listed. Look for a script called ‘palettes’ and click on it to load it in your Code Editor.

    +

    If you scroll down, you will see that the script contains a nested series of dictionaries with lists of hexadecimal color specifications (as described in Chap. F2.1) that describe a color palette, as shown in the code block below. For example, the color palette named “Algae” stored in the cmocean variable consists of seven colors, ranging from dark green to light green (Fig. F6.1.11).

    +

    exports.cmocean = {
    +Thermal: { 7: [‘042333’, ‘2c3395’, ‘744992’, ‘b15f82’, ‘eb7958’, ‘fbb43d’, ‘e8fa5b’ ]
    +},
    +Haline: { 7: [‘2a186c’, ‘14439c’, ‘206e8b’, ‘3c9387’, ‘5ab978’, ‘aad85c’, ‘fdef9a’ ]
    +},
    +Solar: { 7: [‘331418’, ‘682325’, ‘973b1c’, ‘b66413’, ‘cb921a’, ‘dac62f’, ‘e1fd4b’ ]
    +},
    +Ice: { 7: [‘040613’, ‘292851’, ‘3f4b96’, ‘427bb7’, ‘61a8c7’, ‘9cd4da’, ‘eafdfd’ ]
    +},
    +Gray: { 7: [‘000000’, ‘232323’, ‘4a4a49’, ‘727171’, ‘9b9a9a’, ‘cacac9’, ‘fffffd’ ]
    +},
    +Oxy: { 7: [‘400505’, ‘850a0b’, ‘6f6f6e’, ‘9b9a9a’, ‘cbcac9’, ‘ebf34b’, ‘ddaf19’ ]
    +},
    +Deep: { 7: [‘fdfecc’, ‘a5dfa7’, ‘5dbaa4’, ‘488e9e’, ‘3e6495’, ‘3f396c’, ‘281a2c’ ]
    +},
    +Dense: { 7: [‘e6f1f1’, ‘a2cee2’, ‘76a4e5’, ‘7871d5’, ‘7642a5’, ‘621d62’, ‘360e24’ ]
    +},
    +Algae: { 7: [‘d7f9d0’, ‘a2d595’, ‘64b463’, ‘129450’, ‘126e45’, ‘1a482f’, ‘122414’ ]
    +},
    +…
    }

    -

    Notice that the variable is named exports.cmocean. Adding exports to the name of a function or variable makes it available to other scripts to use, as it gets added to a special global variable (Chang 2017).

    -

    -

    Fig. F6.1.11 Some of the color palettes from the ee-palettes GitHub repository

    +

    Notice that the variable is named exports.cmocean. Adding exports to the name of a function or variable makes it available to other scripts to use, as it gets added to a special global variable (Chang 2017).

    +
    +
    +

    +

    Fig. F6.1.11 Some of the color palettes from the ee-palettes GitHub repository

    +
    +

    To see all the color palettes available in this module, go to https://github.com/gee-community/ee-palettes.

    -

    Now let’s try using the ee-palettes module. Look for a script in the same repository called ‘palettes-test’ and click on it to load it in your Code Editor. When you run the script, you will see digital elevation data from the National Aeronautics and Space Administration Shuttle Radar Topography Mission satellite visualized using two palettes,  colorbrewer.Blues and cmocean.Algae. The map will have two layers that show the same data with different palettes.

    -

    The script first imports the digital elevation model data in the Imports section of the Code Editor.

    -

    var dem = ee.Image(‘USGS/SRTMGL1_003’);

    -

    The script then loads the ee-palettes module by using the require function. The path to the module, ‘users/gena/packages:palettes’, is passed to the function. The require function is then stored in a variable named ‘palettes’, which will be used later to obtain the palettes for data visualization.

    -

    var palettes = require(‘users/gena/packages:palettes’);

    -

    As described by Donchyts et al. (2019), “Each palette is defined by a group and a name, which are separated by a period (JS object dot notation), and a color level. To retrieve a desired palette, use JS object notation to specify the group, name, and number of color levels.” We define the color palette Algae as palettes.cmocean.Algae[7] because it is part of the group cmocean and has 7 color levels. In the next code block, you can see that the palettes (i.e., lists of hex colors) have been defined for use by setting them as the value for the palette key in the visParams object supplied to the Map.addLayer function.

    -

    // colorbrewer
    -Map.addLayer(dem, {
    -   min: 0,
    -   max: 3000,
    -   palette: palettes.colorbrewer.Blues[9]
    -}, ‘colorbrewer Blues[9]’);

    -

    // cmocean
    -Map.addLayer(dem, {
    -   min: 0,
    -   max: 3000,
    -   palette: palettes.cmocean.Algae[7]
    -}, ‘cmocean Algae[7]’);

    -

    Question 8. Try adding a third layer to the Map with a different color palette from ee-palettes. How easy was it to do?

    +

    Now let’s try using the ee-palettes module. Look for a script in the same repository called ‘palettes-test’ and click on it to load it in your Code Editor. When you run the script, you will see digital elevation data from the National Aeronautics and Space Administration Shuttle Radar Topography Mission satellite visualized using two palettes, colorbrewer.Blues and cmocean.Algae. The map will have two layers that show the same data with different palettes.

    +

    The script first imports the digital elevation model data in the Imports section of the Code Editor.

    +

    var dem = ee.Image(‘USGS/SRTMGL1_003’);

    +

    The script then loads the ee-palettes module by using the require function. The path to the module, ‘users/gena/packages:palettes’, is passed to the function. The require function is then stored in a variable named ‘palettes’, which will be used later to obtain the palettes for data visualization.

    +

    var palettes = require(‘users/gena/packages:palettes’);

    +

    As described by Donchyts et al. (2019), “Each palette is defined by a group and a name, which are separated by a period (JS object dot notation), and a color level. To retrieve a desired palette, use JS object notation to specify the group, name, and number of color levels.” We define the color palette Algae as palettes.cmocean.Algae[7] because it is part of the group cmocean and has 7 color levels. In the next code block, you can see that the palettes (i.e., lists of hex colors) have been defined for use by setting them as the value for the palette key in the visParams object supplied to the Map.addLayer function.

    +
    // colorbrewer  
+Map.addLayer(dem, {  
+   min: 0,  
+   max: 3000,  
+   palette: palettes.colorbrewer.Blues[9]  
+}, 'colorbrewer Blues[9]');  
+  
+// cmocean  
+Map.addLayer(dem, {  
+   min: 0,  
+   max: 3000,  
+   palette: palettes.cmocean.Algae[7]  
+}, 'cmocean Algae[7]');
    +

    Question 8. Try adding a third layer to the Map with a different color palette from ee-palettes. How easy was it to do?

    Now that you have loaded and used a module shared by someone else, you can try your hand at creating your own module and sharing it with someone else in your group. First, go to the shared repository that you created in Sect. 3, create a new script in that repository, and name it “cloudmasking.”

    -

    Then, go to the Examples repository at the bottom of the Scripts tab and select a function from the Cloud Masking repository. Let’s use the Landsat8 Surface Reflectance cloud masking script as an example. In that script, you will see the code shown in the block below. Copy all of it into your empty script.  

    -

    // This example demonstrates the use of the Landsat 8 Collection 2, Level 2
    -// QA_PIXEL band (CFMask) to mask unwanted pixels.
    -function maskL8sr(image) {   // Bit 0 - Fill   // Bit 1 - Dilated Cloud   // Bit 2 - Cirrus   // Bit 3 - Cloud   // Bit 4 - Cloud Shadow   var qaMask = image.select(‘QA_PIXEL’).bitwiseAnd(parseInt(‘11111’,       2)).eq(0);   var saturationMask = image.select(‘QA_RADSAT’).eq(0);   // Apply the scaling factors to the appropriate bands.   var opticalBands = image.select(‘SR_B.’).multiply(0.0000275).add(-       0.2);   var thermalBands = image.select(’ST_B.*’).multiply(0.00341802)
    -       .add(149.0);   // Replace the original bands with the scaled ones and apply the masks.   return image.addBands(opticalBands, null, true)
    -       .addBands(thermalBands, null, true)
    -       .updateMask(qaMask)
    -       .updateMask(saturationMask);
    -}

    -

    // Map the function over one year of data.
    -var collection = ee.ImageCollection(‘LANDSAT/LC08/C02/T1_L2’)
    -   .filterDate(‘2020-01-01’, ‘2021-01-01’)
    -   .map(maskL8sr);

    -

    var composite = collection.median();

    -

    // Display the results.
    -Map.setCenter(-4.52, 40.29, 7); // Iberian Peninsula
    -Map.addLayer(composite, {
    -   bands: [‘SR_B4’, ‘SR_B3’, ‘SR_B2’],
    -   min: 0,
    -   max: 0.3
    -});

    +

    Then, go to the Examples repository at the bottom of the Scripts tab and select a function from the Cloud Masking repository. Let’s use the Landsat8 Surface Reflectance cloud masking script as an example. In that script, you will see the code shown in the block below. Copy all of it into your empty script.

    +
    // This example demonstrates the use of the Landsat 8 Collection 2, Level 2  
+// QA_PIXEL band (CFMask) to mask unwanted pixels.  
+function maskL8sr(image) {   // Bit 0 - Fill   // Bit 1 - Dilated Cloud   // Bit 2 - Cirrus   // Bit 3 - Cloud   // Bit 4 - Cloud Shadow   var qaMask = image.select('QA_PIXEL').bitwiseAnd(parseInt('11111',       2)).eq(0);   var saturationMask = image.select('QA_RADSAT').eq(0);   // Apply the scaling factors to the appropriate bands.   var opticalBands = image.select('SR_B.').multiply(0.0000275).add(-       0.2);   var thermalBands = image.select('ST_B.*').multiply(0.00341802)  
+       .add(149.0);   // Replace the original bands with the scaled ones and apply the masks.   return image.addBands(opticalBands, null, true)  
+       .addBands(thermalBands, null, true)  
+       .updateMask(qaMask)  
+       .updateMask(saturationMask);  
+}  
+  
+// Map the function over one year of data.  
+var collection = ee.ImageCollection('LANDSAT/LC08/C02/T1_L2')  
+   .filterDate('2020-01-01', '2021-01-01')  
+   .map(maskL8sr);  
+  
+var composite = collection.median();  
+  
+// Display the results.  
+Map.setCenter(-4.52, 40.29, 7); // Iberian Peninsula  
+Map.addLayer(composite, {  
+   bands: ['SR_B4', 'SR_B3', 'SR_B2'],  
+   min: 0,  
+   max: 0.3  
+});

    Note that this code is well commented and has a header that describes what the script does. Don’t forget to comment your code and describe what you are doing each step of the way. This is a good practice for collaborative coding and for your own future reference.

    -

    Imagine that you changed this maskL8sr function slightly for some reason and want to make it available to other users and scripts. To do that, you can turn the function into a module. Copy and modify the code from the example code into the new script you created called “cloudmasking.” (Hint: Store the function in a variable starting with exports. Be careful that you don’t accidentally use Export, which is used to export datasets.)

    +

    Imagine that you changed this maskL8sr function slightly for some reason and want to make it available to other users and scripts. To do that, you can turn the function into a module. Copy and modify the code from the example code into the new script you created called “cloudmasking.” (Hint: Store the function in a variable starting with exports. Be careful that you don’t accidentally use Export, which is used to export datasets.)

    Your script should be similar to the following code.

    -

    exports.maskL8sr = function(image) {   // Bit 0 - Fill   // Bit 1 - Dilated Cloud   // Bit 2 - Cirrus   // Bit 3 - Cloud   // Bit 4 - Cloud Shadow   var qaMask = image.select(‘QA_PIXEL’).bitwiseAnd(parseInt(       ‘11111’, 2)).eq(0);   var saturationMask = image.select(‘QA_RADSAT’).eq(0);   // Apply the scaling factors to the appropriate bands.   var opticalBands = image.select(‘SR_B.’).multiply(0.0000275)
    -       .add(-0.2);   var thermalBands = image.select(’ST_B.*’).multiply(0.00341802)
    -       .add(149.0);   // Replace the original bands with the scaled ones and apply the masks.   return image.addBands(opticalBands, null, true)
    -       .addBands(thermalBands, null, true)
    -       .updateMask(qaMask)
    -       .updateMask(saturationMask);
    +

    exports.maskL8sr = function(image) { // Bit 0 - Fill // Bit 1 - Dilated Cloud // Bit 2 - Cirrus // Bit 3 - Cloud // Bit 4 - Cloud Shadow var qaMask = image.select(‘QA_PIXEL’).bitwiseAnd(parseInt( ‘11111’, 2)).eq(0); var saturationMask = image.select(‘QA_RADSAT’).eq(0); // Apply the scaling factors to the appropriate bands. var opticalBands = image.select(‘SR_B.’).multiply(0.0000275)
    +.add(-0.2); var thermalBands = image.select(’ST_B.*’).multiply(0.00341802)
    +.add(149.0); // Replace the original bands with the scaled ones and apply the masks. return image.addBands(opticalBands, null, true)
    +.addBands(thermalBands, null, true)
    +.updateMask(qaMask)
    +.updateMask(saturationMask);
    }

    Next, you will create a test script that makes use of the cloud masking module you just made. Begin by creating a new script in your shared repository called “cloudmasking-test.” You can modify the last part of the example cloud masking script to use your module.

    -

    // Map the function over one year of data.
    -var collection = ee.ImageCollection(‘LANDSAT/LC08/C02/T1_L2’)
    -   .filterDate(‘2020-01-01’, ‘2021-01-01’)
    -   .map(maskL8sr);

    -

    var composite = collection.median();

    -

    // Display the results.
    -Map.setCenter(-4.52, 40.29, 7); // Iberian Peninsula
    -Map.addLayer(composite, {
    -   bands: [‘SR_B4’, ‘SR_B3’, ‘SR_B2’],
    -   min: 0,
    -   max: 0.3
    -});

    -

    Question 9. How will you modify the cloud masking script to use your module? What does the script look like?

    -

    Answer: Your code might look something like the code block below.

    -

    // Load the module
    -var myCloudFunctions = require(   ‘users/myusername/my-shared-repo:cloudmasking’);

    -

    // Map the function over one year of data.
    -var collection = ee.ImageCollection(‘LANDSAT/LC08/C02/T1_L2’)
    -   .filterDate(‘2020-01-01’, ‘2021-01-01’)
    -   .map(myCloudFunctions.maskL8sr);

    -

    var composite = collection.median();

    -

    // Display the results.
    -Map.setCenter(-4.52, 40.29, 7); // Iberian Peninsula
    -Map.addLayer(composite, {
    -   bands: [‘SR_B4’, ‘SR_B3’, ‘SR_B2’],
    -   min: 0,
    -   max: 0.3
    -});

    +
    // Map the function over one year of data.  
+var collection = ee.ImageCollection('LANDSAT/LC08/C02/T1_L2')  
+   .filterDate('2020-01-01', '2021-01-01')  
+   .map(maskL8sr);  
+  
+var composite = collection.median();  
+  
+// Display the results.  
+Map.setCenter(-4.52, 40.29, 7); // Iberian Peninsula  
+Map.addLayer(composite, {  
+   bands: ['SR_B4', 'SR_B3', 'SR_B2'],  
+   min: 0,  
+   max: 0.3  
+});
    +

    Question 9. How will you modify the cloud masking script to use your module? What does the script look like?

    +

    Answer: Your code might look something like the code block below.

    +
    // Load the module  
+var myCloudFunctions = require(   'users/myusername/my-shared-repo:cloudmasking');  
+  
+// Map the function over one year of data.  
+var collection = ee.ImageCollection('LANDSAT/LC08/C02/T1_L2')  
+   .filterDate('2020-01-01', '2021-01-01')  
+   .map(myCloudFunctions.maskL8sr);  
+  
+var composite = collection.median();  
+  
+// Display the results.  
+Map.setCenter(-4.52, 40.29, 7); // Iberian Peninsula  
+Map.addLayer(composite, {  
+   bands: ['SR_B4', 'SR_B3', 'SR_B2'],  
+   min: 0,  
+   max: 0.3  
+});

    Synthesis

    @@ -1242,7 +1439,7 @@ Chapter Information

    Author

    -

    Jillian M. Deines, Stefania Di Tommaso, Nicholas Clinton, Noel Gorelick    

    +

    Jillian M. Deines, Stefania Di Tommaso, Nicholas Clinton, Noel Gorelick

    Overview

    @@ -1263,8 +1460,8 @@ Chapter Information
  • Import images and image collections, filter, and visualize (Part F1).
  • Write a function and map it over an ImageCollection (Chap. F4.0).
  • Export and import results as Earth Engine assets (Chap. F5.0).
    • -
  • Understand distinctions among Image, ImageCollection, Feature and FeatureCollection Earth Engine objects (Part F1, Part F2, Part F5).
    • -
  • Use the require function to load code from existing modules (Chap. F6.1).
    • +
  • Understand distinctions among Image, ImageCollection, Feature and FeatureCollection Earth Engine objects (Part F1, Part F2, Part F5).
    • +
  • Use the require function to load code from existing modules (Chap. F6.1).
    • @@ -1283,16 +1480,16 @@ Chapter Information

    10.0.1 Earth Engine: Under the Hood

    As you use Earth Engine, you may begin to have questions about how it works and how you can use that knowledge to optimize your workflow. In general, the inner workings are opaque to users. Typical fixes and approaches that data scientists use to manage memory constraints often don’t apply. It’s helpful to know what users can and cannot control, and how your scripts translate to Earth Engine’s server operations.

    Earth Engine is a parallel, distributed system (see Gorelick et al. 2017), which means that when you submit tasks, it breaks up pieces of your query onto different processors to complete them more efficiently. It then collects the results and returns them to you. For many users, not having to manually design this parallel, distributed processing is a huge benefit. For some advanced users, it can be frustrating to not have better control. We’d argue that leaving the details up to Earth Engine is a huge time-saver for most cases, and learning to work within a few constraints is a good time investment.

    -

    One core concept useful to master is the relationship between client-side and server-side operations. Client-side operations are performed within your browser (for the JavaScript API Code Editor) or local system (for the Python API). These include things such as manipulating strings or numbers in JavaScript. Server-side operations are executed on Google’s servers and include all of the ee.* functions. By using the Earth Engine APIs—JavaScript or Python—you are building a chain of commands to send to the servers and later receive the result back. As much as possible, you want to structure your code to send all the heavy lifting to Google, and keep processing off of your local resources.

    -

    In other words, your work in the Code Editor is making a description of a computation. All ee objects are just placeholders for server-side objects—their actual value does not exist locally on your computer. To see or use the actual value, it has to be evaluated by the server. If you print an Earth Engine object, it calls getInfo to evaluate and return the value. In contrast, you can also work with JavaScript/Python lists or numbers locally, and do basic JavaScript/Python things to them, like add numbers together or loop over items. These are client-side objects. Whenever you bring a server-side object into your local environment, there’s a computational cost.

    +

    One core concept useful to master is the relationship between client-side and server-side operations. Client-side operations are performed within your browser (for the JavaScript API Code Editor) or local system (for the Python API). These include things such as manipulating strings or numbers in JavaScript. Server-side operations are executed on Google’s servers and include all of the ee.* functions. By using the Earth Engine APIs—JavaScript or Python—you are building a chain of commands to send to the servers and later receive the result back. As much as possible, you want to structure your code to send all the heavy lifting to Google, and keep processing off of your local resources.

    +

    In other words, your work in the Code Editor is making a description of a computation. All ee objects are just placeholders for server-side objects—their actual value does not exist locally on your computer. To see or use the actual value, it has to be evaluated by the server. If you print an Earth Engine object, it calls getInfo to evaluate and return the value. In contrast, you can also work with JavaScript/Python lists or numbers locally, and do basic JavaScript/Python things to them, like add numbers together or loop over items. These are client-side objects. Whenever you bring a server-side object into your local environment, there’s a computational cost.

    Table F6.2.1 describes some nuts and bolts about Earth Engine and their implications. Table F6.2.2 provides some of the existing limits on individual tasks.

    -

    Table F6.2.1 Characterics of Google Earth Engine and implications for running large jobs 

    +

    Table F6.2.1 Characterics of Google Earth Engine and implications for running large jobs

    Earth Engine characteristics

    Implications

    A parallel, distributed system

    Occasionally, doing the exact same thing in two different orders can result in different processing distributions, impacting the ability to complete the task within system limits.

    Most processing is done per tile (generally a square that is 256 x 256 pixels).

    -

    Tasks that require many tiles are the most memory intensive. Some functions have a tileScale argument that reduces tile size, allowing processing-intensive jobs to succeed (at the cost of reduced speed).

    +

    Tasks that require many tiles are the most memory intensive. Some functions have a tileScale argument that reduces tile size, allowing processing-intensive jobs to succeed (at the cost of reduced speed).

    Export mode has higher memory and time allocations than interactive mode.

    It’s better to export large jobs. You can export to your Earth Engine assets, your Google Drive, or Google Cloud Storage.

    Some operations are cached temporarily.

    @@ -1301,21 +1498,21 @@ Chapter Information

    There’s a hard limit on data size for any individual server; large computations need to be done in parallel using Earth Engine functions.

    The image processing domain, scale, and projection are defined by the specified output and applied backwards throughout the processing chain.

    There are not many cases when you will need to manually reproject images, and these operations are costly. Similarly, manually “clipping” images is typically unnecessary.

    -

    Table F6.2.2 Size limits for Earth Engine tasks

    +

    Table F6.2.2 Size limits for Earth Engine tasks

    Earth Engine Component

    Limits

    Interactive mode

    Can print up to 5000 records. Computations must finish within five minutes.

    Export mode

    -

    Jobs have no time limit as long as they continue to make reasonable progress (defined roughly as 600 seconds per feature, two hours per aggregation, and 10 minutes per tile). If any one tile, feature, or aggregation takes too long, the whole job will get canceled. Any jobs that take longer than one week to run will likely fail due to Earth Engine’s software update release cycles.

    +

    Jobs have no time limit as long as they continue to make reasonable progress (defined roughly as 600 seconds per feature, two hours per aggregation, and 10 minutes per tile). If any one tile, feature, or aggregation takes too long, the whole job will get canceled. Any jobs that take longer than one week to run will likely fail due to Earth Engine’s software update release cycles.

    Table assets

    Maximum of 100 million features, 1000 properties (columns), and 100,000 vertices for a geometry.

    10.0.2 The Importance of Coding Best Practices

    Good code scales better than bad code. But what is good code? Generally, for Earth Engine, good code means 1) using Earth Engine’s server-side operators; 2) avoiding multiple passes through the same image collection; 3) avoiding unnecessary conversions; and 4) setting the processing scale or sample numbers appropriate for your use case, i.e., avoid using very fine scales or large samples without reason.

    -

    We encourage readers to become familiar with the “Coding Best Practices” page in the online Earth Engine User Guide. This page provides examples for avoiding mixing client- and server-side functions, unnecessary conversions, costly algorithms, combining reducers, and other helpful tips. Similarly, the “Debugging Guide–Scaling Errors” page of the online Earth Engine User Guide covers some common problems and solutions.

    -

    In addition, some Earth Engine functions are more efficient than others. For example, Image.reduceRegions is more efficient than Image.sampleRegions, because sampleRegions regenerates the geometries under the hood. These types of best practices are trickier to enumerate and somewhat idiosyncratic. We encourage users to learn about and make use of the Profiler tab, which will track and display the resources used for each operation within your script. This can help identify areas to focus efficiency improvements. Note that the profiler itself increases resource use, so only use it when necessary to develop a script and remove it for production-level execution. Other ways to discover best practices include following/posting questions to GIS StackExchange or the Earth Engine Developer’s Discussion Group, swapping code with others, and experimentation.

    +

    We encourage readers to become familiar with the “Coding Best Practices” page in the online Earth Engine User Guide. This page provides examples for avoiding mixing client- and server-side functions, unnecessary conversions, costly algorithms, combining reducers, and other helpful tips. Similarly, the “Debugging Guide–Scaling Errors” page of the online Earth Engine User Guide covers some common problems and solutions.

    +

    In addition, some Earth Engine functions are more efficient than others. For example, Image.reduceRegions is more efficient than Image.sampleRegions, because sampleRegions regenerates the geometries under the hood. These types of best practices are trickier to enumerate and somewhat idiosyncratic. We encourage users to learn about and make use of the Profiler tab, which will track and display the resources used for each operation within your script. This can help identify areas to focus efficiency improvements. Note that the profiler itself increases resource use, so only use it when necessary to develop a script and remove it for production-level execution. Other ways to discover best practices include following/posting questions to GIS StackExchange or the Earth Engine Developer’s Discussion Group, swapping code with others, and experimentation.

    @@ -1324,73 +1521,89 @@ Chapter Information

    10.1.1 1.1. Scaling Up with Earth Engine Operators: Annual Daily Climate Data

    Earth Engine’s operators are designed to parallelize queries on the backend without user intervention. In many cases, they are sufficient to accomplish a scaling operation.

    -

    As an example, we will extract a daily time series of precipitation, maximum temperature, and minimum temperature for county polygons in the United States. We will use the GRIDMET Climate Reanalysis product (Abatzoglou 2013), which provides daily, 4000 m resolution gridded meteorological data from 1979 to the present across the contiguous United States. To save time for this practicum, we will focus on the states of Indiana, Illinois, and Iowa in the central United States, which together include 293 counties (Fig. F6.2.1).

    -

    -

    Fig. F6.2.1 Map of study area, showing 293 county features within the states of Iowa, Illinois, and Indiana in the United States

    -

    This example uses the ee.Image.reduceRegions operator, which extracts statistics from an Image for each Feature (point or polygon) in a FeatureCollection. We will map the reduceRegions operator over each daily image in an ImageCollection, thus providing us with the daily climate information for each county of interest.

    +

    As an example, we will extract a daily time series of precipitation, maximum temperature, and minimum temperature for county polygons in the United States. We will use the GRIDMET Climate Reanalysis product (Abatzoglou 2013), which provides daily, 4000 m resolution gridded meteorological data from 1979 to the present across the contiguous United States. To save time for this practicum, we will focus on the states of Indiana, Illinois, and Iowa in the central United States, which together include 293 counties (Fig. F6.2.1).

    +
    +
    +

    +

    Fig. F6.2.1 Map of study area, showing 293 county features within the states of Iowa, Illinois, and Indiana in the United States

    +
    +
    +

    This example uses the ee.Image.reduceRegions operator, which extracts statistics from an Image for each Feature (point or polygon) in a FeatureCollection. We will map the reduceRegions operator over each daily image in an ImageCollection, thus providing us with the daily climate information for each county of interest.

    Note that although our example uses a climate ImageCollection, this approach transfers to any ImageCollection, including satellite imagery, as well as image collections that you have already processed, such as cloud masking (Chap. F4.3) or time series aggregation (Chap. F4.2).

    First, define the FeatureCollection, ImageCollection, and time period:

    -

    // Load county dataset.
    -// Filter counties in Indiana, Illinois, and Iowa by state FIPS code.
    -// Select only the unique ID column for simplicity.
    -var countiesAll = ee.FeatureCollection(‘TIGER/2018/Counties’);
    -var states = [‘17’, ‘18’, ‘19’];
    -var uniqueID = ‘GEOID’;
    -var featColl = countiesAll.filter(ee.Filter.inList(‘STATEFP’, states))
    -   .select(uniqueID);

    -

    print(featColl.size());
    -print(featColl.limit(1));

    -

    // Visualize target features (create Figure F6.2.1).
    -Map.centerObject(featColl, 5);
    -Map.addLayer(featColl);

    -

    // specify years of interest
    -var startYear = 2020;
    -var endYear = 2020;

    -

    // climate dataset info
    -var imageCollectionName = ‘IDAHO_EPSCOR/GRIDMET’;
    -var bandsWanted = [‘pr’, ‘tmmn’, ‘tmmx’];
    -var scale = 4000;

    -

    Printing the size of the FeatureCollection indicates that there are 293 counties in our subset. Since we want to pull a daily time series for one year, our final dataset will have 106,945 rows—one for each county-day.

    -

    Note that from our county FeatureCollection, we select only the GEOID column, which represents a unique identifier for each record in this dataset. We do this here to simplify print outputs; we could also specify which properties to include in the export function (see below).

    -

    Next, load and filter the climate data. Note we adjust the end date to January 1 of the following year, rather than December 31 of the specified year, since the filterDate function has an inclusive start date argument and an exclusive end date argument; without this modification the output would lack data for December 31.

    -

    // Load and format climate data.
    -var startDate = startYear + ‘-01-01’;

    -

    var endYear_adj = endYear + 1;
    -var endDate = endYear_adj + ‘-01-01’;

    -

    var imageCollection = ee.ImageCollection(imageCollectionName)
    -   .select(bandsWanted)
    -   .filterBounds(featColl)
    -   .filterDate(startDate, endDate);

    -

    Now get the mean value for each climate attribute within each county feature. Here, we map the ee.Image.reduceRegions call over the ImageCollection, specifying an ee.Reducer.mean reducer. The reducer will apply to each band in the image, and it returns the FeatureCollection with new properties. We also add a ‘date_ymd’ time property extracted from the image to correctly associate daily values with their date. Finally, we flatten the output to reform a single FeatureCollection with one feature per county-day.

    -

    // get values at features
    -var sampledFeatures = imageCollection.map(function(image) {   return image.reduceRegions({
    -           collection: featColl,
    -           reducer: ee.Reducer.mean(),
    -           scale: scale
    -       }).filter(ee.Filter.notNull(
    -       bandsWanted)) // drop rows with no data       .map(function(f) { // add date property           var time_start = image.get(               ‘system:time_start’);           var dte = ee.Date(time_start).format(               ‘YYYYMMdd’);           return f.set(‘date_ymd’, dte);
    -       });
    -}).flatten();

    -

    print(sampledFeatures.limit(1));

    -

    Note that we include a filter to remove feature-day rows that lacked data. While this is less common when using gridded climate products, missing data can be common when reducing satellite images. This is because satellite collections come in scene tiles, and each image tile likely does not overlap all of our features unless it has first been aggregated temporally. It can also occur if a cloud mask has been applied to an image prior to the reduction. By filtering out null values, we can reduce empty rows.

    -

    Now explore the result. If we simply print(sampledFeatures) we get our first error message: “User memory limit exceeded.” This is because we’ve created a FeatureCollection that exceeds the size limits set for interactive mode. How many are there? We could try print(sampledFeatures.size()), but due to the larger size, we receive a “Computation timed out” message—it’s unable to tell us. Of course, we know that we expect 293 counties x 365 days = 106,945 features. We can, however, check that our reducer has worked as expected by asking Earth Engine for just one feature: print(sampledFeatures.limit(1)).

    -

    -

    Fig. F6.2.2 Screenshot of the print output for one feature after the reduceRegions call

    +
    // Load county dataset.  
+// Filter counties in Indiana, Illinois, and Iowa by state FIPS code.  
+// Select only the unique ID column for simplicity.  
+var countiesAll = ee.FeatureCollection('TIGER/2018/Counties');  
+var states = ['17', '18', '19'];  
+var uniqueID = 'GEOID';  
+var featColl = countiesAll.filter(ee.Filter.inList('STATEFP', states))  
+   .select(uniqueID);  
+  
+print(featColl.size());  
+print(featColl.limit(1));  
+  
+// Visualize target features (create Figure F6.2.1).  
+Map.centerObject(featColl, 5);  
+Map.addLayer(featColl);  
+  
+// specify years of interest  
+var startYear = 2020;  
+var endYear = 2020;  
+  
+// climate dataset info  
+var imageCollectionName = 'IDAHO_EPSCOR/GRIDMET';  
+var bandsWanted = ['pr', 'tmmn', 'tmmx'];  
+var scale = 4000;
    +

    Printing the size of the FeatureCollection indicates that there are 293 counties in our subset. Since we want to pull a daily time series for one year, our final dataset will have 106,945 rows—one for each county-day.

    +

    Note that from our county FeatureCollection, we select only the GEOID column, which represents a unique identifier for each record in this dataset. We do this here to simplify print outputs; we could also specify which properties to include in the export function (see below).

    +

    Next, load and filter the climate data. Note we adjust the end date to January 1 of the following year, rather than December 31 of the specified year, since the filterDate function has an inclusive start date argument and an exclusive end date argument; without this modification the output would lack data for December 31.

    +
    // Load and format climate data.  
+var startDate = startYear + '-01-01';  
+  
+var endYear_adj = endYear + 1;  
+var endDate = endYear_adj + '-01-01';  
+  
+var imageCollection = ee.ImageCollection(imageCollectionName)  
+   .select(bandsWanted)  
+   .filterBounds(featColl)  
+   .filterDate(startDate, endDate);
    +

    Now get the mean value for each climate attribute within each county feature. Here, we map the ee.Image.reduceRegions call over the ImageCollection, specifying an ee.Reducer.mean reducer. The reducer will apply to each band in the image, and it returns the FeatureCollection with new properties. We also add a ‘date_ymd’ time property extracted from the image to correctly associate daily values with their date. Finally, we flatten the output to reform a single FeatureCollection with one feature per county-day.

    +
    // get values at features  
+var sampledFeatures = imageCollection.map(function(image) {   return image.reduceRegions({  
+           collection: featColl,  
+           reducer: ee.Reducer.mean(),  
+           scale: scale  
+       }).filter(ee.Filter.notNull(  
+       bandsWanted)) // drop rows with no data       .map(function(f) { // add date property           var time_start = image.get(               'system:time_start');           var dte = ee.Date(time_start).format(               'YYYYMMdd');           return f.set('date_ymd', dte);  
+       });  
+}).flatten();  
+  
+print(sampledFeatures.limit(1));
    +

    Note that we include a filter to remove feature-day rows that lacked data. While this is less common when using gridded climate products, missing data can be common when reducing satellite images. This is because satellite collections come in scene tiles, and each image tile likely does not overlap all of our features unless it has first been aggregated temporally. It can also occur if a cloud mask has been applied to an image prior to the reduction. By filtering out null values, we can reduce empty rows.

    +

    Now explore the result. If we simply print(sampledFeatures) we get our first error message: “User memory limit exceeded.” This is because we’ve created a FeatureCollection that exceeds the size limits set for interactive mode. How many are there? We could try print(sampledFeatures.size()), but due to the larger size, we receive a “Computation timed out” message—it’s unable to tell us. Of course, we know that we expect 293 counties x 365 days = 106,945 features. We can, however, check that our reducer has worked as expected by asking Earth Engine for just one feature: print(sampledFeatures.limit(1)).

    +
    +
    +

    +

    Fig. F6.2.2 Screenshot of the print output for one feature after the reduceRegions call

    +
    +

    Here, we can see the precipitation, minimum temperature, and maximum temperature for the county with GEOID = 17121 on January 1, 2020 (Fig. F6.2.2; note temperature is in Kelvin units).

    -

    Next, export the full FeatureCollection as a CSV to a folder in your Google Drive. Specify the names of properties to include. Build part of the filename dynamically based on arguments used for year and data scale, so we don’t need to manually modify the filenames.

    -

    // export info
    -var exportFolder = ‘GEE_scalingUp’;
    -var filename = ‘Gridmet_counties_IN_IL_IA_’ + scale + ‘m_’ +
    -   startYear + ‘-’ + endYear;// prepare export: specify properties/columns to include
    -var columnsWanted = [uniqueID].concat([‘date_ymd’], bandsWanted);
    -print(columnsWanted);

    -

    Export.table.toDrive({
    -   collection: sampledFeatures,
    -   description: filename,
    -   folder: exportFolder,
    -   fileFormat: ‘CSV’,
    -   selectors: columnsWanted
    -});

    +

    Next, export the full FeatureCollection as a CSV to a folder in your Google Drive. Specify the names of properties to include. Build part of the filename dynamically based on arguments used for year and data scale, so we don’t need to manually modify the filenames.

    +
    // export info  
+var exportFolder = 'GEE_scalingUp';  
+var filename = 'Gridmet_counties_IN_IL_IA_' + scale + 'm_' +  
+   startYear + '-' + endYear;// prepare export: specify properties/columns to include  
+var columnsWanted = [uniqueID].concat(['date_ymd'], bandsWanted);  
+print(columnsWanted);  
+  
+Export.table.toDrive({  
+   collection: sampledFeatures,  
+   description: filename,  
+   folder: exportFolder,  
+   fileFormat: 'CSV',  
+   selectors: columnsWanted  
+});
    @@ -1401,156 +1614,175 @@ Note
    -

    Code Checkpoint F62a. The book’s repository contains a script that shows what your code should look like at this point.

    +

    Code Checkpoint F62a. The book’s repository contains a script that shows what your code should look like at this point.

    On our first export, this job took about eight minutes to complete, producing a dataset 6.8 MB in size. The data is ready for downstream use but may need formatting to suit the user’s goals. You can see what the exported CSV looks like in Fig. F6.2.3.

    -

    -

    Fig. F6.2.3 Top six rows of the exported CSV viewed in Microsoft Excel and sorted by county GEOID 

    +
    +
    +

    +

    Fig. F6.2.3 Top six rows of the exported CSV viewed in Microsoft Excel and sorted by county GEOID

    +
    +

    Using the Selectors Argument

    -

    There are two excellent reasons to use the selectors argument in your Export.table.toDrive call. First, if the argument is not specified, Earth Engine will generate the column names for the exported CSV from the first feature in your FeatureCollection. If that feature is missing properties, those properties will be dropped from the export for all features.

    -

    Perhaps even more important if you are seeking to scale up an analysis, including unnecessary columns can greatly increase file size and even processing time. For example, Earth Engine includes a .geo field that contains a GeoJSON description of each spatial feature. For non-simple geometries, the field can be quite large, as it lists coordinates for each polygon vertex. For many purposes, it’s not necessary to include this information for each daily record (here, 365 daily rows per feature).

    -

    For example, when we ran the same job as above but did not use the selectors argument, the output dataset was 5.7 GB (versus 6.8 MB!) and the runtime was slower. This is a cumbersomely large file, with no real benefit. We generally recommend dropping the .geo column and other unnecessary properties. To retain spatial information, a unique identifier for each feature can be used for downstream joins with the spatial data or other properties. If working with point data, latitude and longitude columns can be added prior to export to maintain easily accessible geographic information, although the .geo column for point data is far smaller than for irregularly shaped polygon features.

    +

    There are two excellent reasons to use the selectors argument in your Export.table.toDrive call. First, if the argument is not specified, Earth Engine will generate the column names for the exported CSV from the first feature in your FeatureCollection. If that feature is missing properties, those properties will be dropped from the export for all features.

    +

    Perhaps even more important if you are seeking to scale up an analysis, including unnecessary columns can greatly increase file size and even processing time. For example, Earth Engine includes a .geo field that contains a GeoJSON description of each spatial feature. For non-simple geometries, the field can be quite large, as it lists coordinates for each polygon vertex. For many purposes, it’s not necessary to include this information for each daily record (here, 365 daily rows per feature).

    +

    For example, when we ran the same job as above but did not use the selectors argument, the output dataset was 5.7 GB (versus 6.8 MB!) and the runtime was slower. This is a cumbersomely large file, with no real benefit. We generally recommend dropping the .geo column and other unnecessary properties. To retain spatial information, a unique identifier for each feature can be used for downstream joins with the spatial data or other properties. If working with point data, latitude and longitude columns can be added prior to export to maintain easily accessible geographic information, although the .geo column for point data is far smaller than for irregularly shaped polygon features.

    10.1.2 1.2. Scaling Across Time by Batching: Get 20 Years of Daily Climate Data

    Above, we extracted one year of daily data for our 293 counties. Let’s say we want to do the same thing, but for 2001–2020. We have already written our script to flexibly specify years, so it’s fairly adaptable to this new use case:

    -

    // specify years of interest
    -var startYear = 2020;
    -var endYear = 2020;

    -

    If we only wanted a few years for a small number of features, we could just modify the startYear or endYear and proceed. Indeed, our current example is modest in size and number of features, and we were able to run 2001–2020 in one export job that took about two hours, with an output file size of 299 MB. However, with larger feature collections, or hourly data, we will again start to bump up against Earth Engine’s limits. Generally, jobs of this sort do not fail quickly—exports are allowed to run as long as they continue making progress (see Table F6.2.2). It’s not uncommon, however, for a large job to take well over 24 hours to run, or even to fail after more than 24 hours of run time, as it accumulates too many records or a single aggregation fails. For users, this can be frustrating.

    +
    // specify years of interest  
+var startYear = 2020;  
+var endYear = 2020;
    +

    If we only wanted a few years for a small number of features, we could just modify the startYear or endYear and proceed. Indeed, our current example is modest in size and number of features, and we were able to run 2001–2020 in one export job that took about two hours, with an output file size of 299 MB. However, with larger feature collections, or hourly data, we will again start to bump up against Earth Engine’s limits. Generally, jobs of this sort do not fail quickly—exports are allowed to run as long as they continue making progress (see Table F6.2.2). It’s not uncommon, however, for a large job to take well over 24 hours to run, or even to fail after more than 24 hours of run time, as it accumulates too many records or a single aggregation fails. For users, this can be frustrating.

    We generally find it simpler to run several small jobs rather than one large job. Outputs can then be combined in external software. This avoids any frustration with long-running jobs or delayed failures, and it allows parts of the task to be run simultaneously. Earth Engine generally executes from 2–20 jobs per user at a time, depending on overall user load (although 20 is rare). As a counterpart, there is some overhead for generating separate jobs.

    Important note: When running a batch of jobs, it may be tempting to use multiple accounts to execute subsets of your batch and thus get your shared results faster. However, doing so is a direct violation of the Earth Engine terms of service and can result in your account(s) being terminated.

    For-Loops: They Are Sometimes OK

    Batching jobs in time is a great way to break up a task into smaller units. Other options include batching jobs by spatial regions defined by polygons (see Sect. 2), or for computationally heavy tasks, batching by both space and time.

    -

    Because Export functions are client-side functions, however, you can’t create an export within an Earth Engine map command. Instead, we need to loop over the variable that will define our batches and create a set of export tasks.

    +

    Because Export functions are client-side functions, however, you can’t create an export within an Earth Engine map command. Instead, we need to loop over the variable that will define our batches and create a set of export tasks.

    But wait! Aren’t we supposed to avoid for-loops at all costs? Yes, within a computational chain. Here, we are using a loop to send multiple computational chains to the server.

    First, we will start with the same script as in Sect. 1.1, but we will modify the start year. We will also modify the desired output filename to be a generic base filename, to which we will append the year for each task within the loop (in the next step).

    -

    // Load county dataset.
    -var countiesAll = ee.FeatureCollection(‘TIGER/2018/Counties’);
    -var states = [‘17’, ‘18’, ‘19’];
    -var uniqueID = ‘GEOID’;
    -var featColl = countiesAll.filter(ee.Filter.inList(‘STATEFP’, states))
    -   .select(uniqueID);

    -

    print(featColl.size());
    -print(featColl.limit(1));
    -Map.addLayer(featColl);

    -

    // Specify years of interest.
    -var startYear = 2001;
    -var endYear = 2020;

    -

    // Climate dataset info.
    -var imageCollectionName = ‘IDAHO_EPSCOR/GRIDMET’;
    -var bandsWanted = [‘pr’, ‘tmmn’, ‘tmmx’];
    -var scale = 4000;

    -

    // Export info.
    -var exportFolder = ‘GEE_scalingUp’;
    -var filenameBase = ‘Gridmet_counties_IN_IL_IA_’ + scale + ‘m_’;

    -

    Now modify the code in Sect. 1.1 to use a looping variable, i, to represent each year. Here, we are using standard JavaScript looping syntax, where i will take on each value between our startYear (2001) and our endYear (2020) for each loop through this section of code, thus creating 20 queries to send to Earth Engine’s servers.

    -

    // Initiate a loop, in which the variable i takes on values of each year.
    -for (var i = startYear; i <= endYear; i++) {        // for each year….  // Load climate collection for that year. var startDate = i + ‘-01-01’;

    -

      var endYear_adj = i + 1; var endDate = endYear_adj + ‘-01-01’; var imageCollection = ee.ImageCollection(imageCollectionName)
    -     .select(bandsWanted)
    -     .filterBounds(featColl)
    -     .filterDate(startDate, endDate); // Get values at feature collection. var sampledFeatures = imageCollection.map(function(image) {   return image.reduceRegions({
    -     collection: featColl,
    -     reducer: ee.Reducer.mean(),        
    -     tileScale: 1,
    -     scale: scale
    -   }).filter(ee.Filter.notNull(bandsWanted))  // remove rows without data     .map(function(f) {                  // add date property       var time_start = image.get(‘system:time_start’);       var dte = ee.Date(time_start).format(‘YYYYMMdd’);       return f.set(‘date_ymd’, dte);
    -   });
    - }).flatten(); // Prepare export: specify properties and filename. var columnsWanted = [uniqueID].concat([‘date_ymd’], bandsWanted); var filename = filenameBase + i; Export.table.toDrive({
    -   collection: sampledFeatures,
    -   description: filename,
    -   folder: exportFolder,
    -   fileFormat: ‘CSV’,
    -   selectors: columnsWanted
    - });
    - 
    -}

    -
    -
    -
    - -
    -
    -Note -
    -
    -
    -

    Code Checkpoint F62b. The book’s repository contains a script that shows what your code should look like at this point.

    -
    -
    -

    When we run this script, it builds our computational query for each year, creating a batch of 20 individual jobs that will show up in the Task pane (Fig. F6.2.4). Each task name includes the year, since we used our looping variable i to modify the base filename we specified.

    -

    -

    Fig. F6.2.4 Creation of batch tasks for each year

    -

    We now encounter a downside to creating batch tasks within the JavaScript Code Editor: we need to click Run to execute each job in turn. Here, we made this easier by programmatically assigning each job the filename we want, so we can hold the Cmd/Ctrl key and click Run to avoid the export task option window and only need to click once per task. Still, one can imagine that at some number of tasks, one’s patience for clicking Run will be exceeded. We assume that number is different for everyone.

    -

    Note: If at any time you have submitted several tasks to the server but want to cancel them all, you can do so more easily from the Earth Engine Task Manager that is linked at the top of the Task pane. You can read about that task manager in the Earth Engine User Guide.

    -

    In order to auto-execute jobs in batch mode, we’d need to use the Python API. Interested users can see the Earth Engine User Guide Python API tutorial for further details about the Python API. 

    -
    -
    -
    -

    10.2 Scaling Across Space via Spatial Tiling

    -

    Breaking up jobs in space is another key strategy for scaling operations in Earth Engine. Here, we will focus on making a cloud-free composite from the Sentinel-2 Level 2A Surface Reflectance product. The approach is similar to that in Chap. F4.3, which explores cloud-free compositing. The main difference is that Landsat scenes come with a reliable quality band for each scene, whereas the process for Sentinel-2 is a bit more complicated and computationally intense (see below).

    -

    Our region of interest is the state of Washington in the United States for demonstration purposes, but the method will work at much larger continental scales as well.

    -

    Cloud Masking Approach

    -

    While we do not intend to cover the theory behind Sentinel-2 cloud masking, we do want to include a brief description of the process to convey the computational needs of this approach.

    -

    The Sentinel-2 Level 2A collection does not come with a robust cloud mask. Instead, we will build one from related products that have been developed for this purpose. Following the existing Sentinel-2 cloud masking tutorials in the Earth Engine guides, this approach requires three Sentinel-2 image collections:

    -
      -
    • The Sentinel-2 Level 2A Surface Reflectance product. This is the dataset we want to use to build our final composite.
      • -
    • The Sentinel-2 Cloud Probability Dataset, an ImageCollection that contains cloud probabilities for each Sentinel-2 scene.
      • -
    • The Sentinel-2 Level 1C top-of-atmosphere product. This collection is needed to run the Cloud Displacement Index to identify cloud shadows, which is calculated using ee.Algorithms.Sentinel2.CDI (see Frantz et al. 2018 for algorithm description).
      • -
    -

    These three image collections all contain 10 m resolution data for every Sentinel-2 scene. We will join them based on their ‘system:index’ property so we can relate each Level 2A scene with the corresponding cloud probability and cloud displacement index. Furthermore, there are two ee.Image.projection steps to control the scale when calculating clouds and their shadows.

    -

    To sum up, the cloud masking approach is computationally costly, thus requiring some thought when applying it at scale.

    -
    -

    10.2.1 2.1. Generate a Cloud-Free Satellite Composite: Limits to On-the-Fly Computing

    -

    Note: Our focus here is on code structure for implementing spatial tiling. Below, we import existing tested functions for cloud masking using the require command.

    -

    First, define our region and time of interest; then, load the module containing the cloud functions.

    -

    // Set the Region of Interest:Seattle, Washington, United States
    -var roi = ee.Geometry.Point([-122.33524518034544, 47.61356183942883]);

    -

    // Dates over which to create a median composite.
    -var start = ee.Date(‘2019-03-01’);
    -var end = ee.Date(‘2019-09-01’);

    -

    // Specify module with cloud mask functions.
    -var s2mask_tools = require(   ‘projects/gee-edu/book:Part F - Fundamentals/F6 - Advanced Topics/F6.2 Scaling Up/modules/s2cloudmask.js’
    -);

    +
    // Load county dataset.  
+var countiesAll = ee.FeatureCollection('TIGER/2018/Counties');  
+var states = ['17', '18', '19'];  
+var uniqueID = 'GEOID';  
+var featColl = countiesAll.filter(ee.Filter.inList('STATEFP', states))  
+   .select(uniqueID);  
+  
+print(featColl.size());  
+print(featColl.limit(1));  
+Map.addLayer(featColl);  
+  
+// Specify years of interest.  
+var startYear = 2001;  
+var endYear = 2020;  
+  
+// Climate dataset info.  
+var imageCollectionName = 'IDAHO_EPSCOR/GRIDMET';  
+var bandsWanted = ['pr', 'tmmn', 'tmmx'];  
+var scale = 4000;  
+  
+// Export info.  
+var exportFolder = 'GEE_scalingUp';  
+var filenameBase = 'Gridmet_counties_IN_IL_IA_' + scale + 'm_';
    +

    Now modify the code in Sect. 1.1 to use a looping variable, i, to represent each year. Here, we are using standard JavaScript looping syntax, where i will take on each value between our startYear (2001) and our endYear (2020) for each loop through this section of code, thus creating 20 queries to send to Earth Engine’s servers.

    +
    // Initiate a loop, in which the variable i takes on values of each year.  
+for (var i = startYear; i <= endYear; i++) {        // for each year....  // Load climate collection for that year. var startDate = i + '-01-01';  
+  
+  var endYear_adj = i + 1; var endDate = endYear_adj + '-01-01'; var imageCollection = ee.ImageCollection(imageCollectionName)  
+     .select(bandsWanted)  
+     .filterBounds(featColl)  
+     .filterDate(startDate, endDate); // Get values at feature collection. var sampledFeatures = imageCollection.map(function(image) {   return image.reduceRegions({  
+     collection: featColl,  
+     reducer: ee.Reducer.mean(),          
+     tileScale: 1,  
+     scale: scale  
+   }).filter(ee.Filter.notNull(bandsWanted))  // remove rows without data     .map(function(f) {                  // add date property       var time_start = image.get('system:time_start');       var dte = ee.Date(time_start).format('YYYYMMdd');       return f.set('date_ymd', dte);  
+   });  
+ }).flatten(); // Prepare export: specify properties and filename. var columnsWanted = [uniqueID].concat(['date_ymd'], bandsWanted); var filename = filenameBase + i; Export.table.toDrive({  
+   collection: sampledFeatures,  
+   description: filename,  
+   folder: exportFolder,  
+   fileFormat: 'CSV',  
+   selectors: columnsWanted  
+ });  
+   
+}
+
+:::{.callout-note}
+Code Checkpoint F62b. The book’s repository contains a script that shows what your code should look like at this point.
+:::
+When we run this script, it builds our computational query for each year, creating a batch of 20 individual jobs that will show up in the Task pane (Fig. F6.2.4). Each task name includes the year, since we used our looping variable i to modify the base filename we specified.
+
+![Fig. F6.2.4 Creation of batch tasks for each year](F6/image63.png)
+
+
+We now encounter a downside to creating batch tasks within the JavaScript Code Editor: we need to click Run to execute each job in turn. Here, we made this easier by programmatically assigning each job the filename we want, so we can hold the Cmd/Ctrl key and click Run to avoid the export task option window and only need to click once per task. Still, one can imagine that at some number of tasks, one’s patience for clicking Run will be exceeded. We assume that number is different for everyone.
+
+Note: If at any time you have submitted several tasks to the server but want to cancel them all, you can do so more easily from the Earth Engine Task Manager that is linked at the top of the Task pane. You can read about that task manager in the Earth Engine User Guide.
+
+In order to auto-execute jobs in batch mode, we’d need to use the Python API. Interested users can see the Earth Engine User Guide Python API tutorial for further details about the Python API. 
+
+## Scaling Across Space via Spatial Tiling
+
+Breaking up jobs in space is another key strategy for scaling operations in Earth Engine. Here, we will focus on making a cloud-free composite from the Sentinel-2 Level 2A Surface Reflectance product. The approach is similar to that in Chap. F4.3, which explores cloud-free compositing. The main difference is that Landsat scenes come with a reliable quality band for each scene, whereas the process for Sentinel-2 is a bit more complicated and computationally intense (see below).
+
+Our region of interest is the state of Washington in the United States for demonstration purposes, but the method will work at much larger continental scales as well.
+
+Cloud Masking Approach
+
+While we do not intend to cover the theory behind Sentinel-2 cloud masking, we do want to include a brief description of the process to convey the computational needs of this approach.
+
+The Sentinel-2 Level 2A collection does not come with a robust cloud mask. Instead, we will build one from related products that have been developed for this purpose. Following the existing Sentinel-2 cloud masking tutorials in the Earth Engine guides, this approach requires three Sentinel-2 image collections:
+
+*   The Sentinel-2 Level 2A Surface Reflectance product. This is the dataset we want to use to build our final composite.
+*   The Sentinel-2 Cloud Probability Dataset, an ImageCollection that contains cloud probabilities for each Sentinel-2 scene.
+*   The Sentinel-2 Level 1C top-of-atmosphere product. This collection is needed to run the Cloud Displacement Index to identify cloud shadows, which is calculated using ee.Algorithms.Sentinel2.CDI (see Frantz et al. 2018 for algorithm description).
+
+These three image collections all contain 10 m resolution data for every Sentinel-2 scene. We will join them based on their 'system:index' property so we can relate each Level 2A scene with the corresponding cloud probability and cloud displacement index. Furthermore, there are two ee.Image.projection steps to control the scale when calculating clouds and their shadows.
+
+To sum up, the cloud masking approach is computationally costly, thus requiring some thought when applying it at scale.
+
+### 2.1. Generate a Cloud-Free Satellite Composite: Limits to On-the-Fly Computing
+
+Note: Our focus here is on code structure for implementing spatial tiling. Below, we import existing tested functions for cloud masking using the require command.
+
+First, define our region and time of interest; then, load the module containing the cloud functions.
+
+// Set the Region of Interest:Seattle, Washington, United States  
+var roi = ee.Geometry.Point([-122.33524518034544, 47.61356183942883]);  
+  
+// Dates over which to create a median composite.  
+var start = ee.Date('2019-03-01');  
+var end = ee.Date('2019-09-01');  
+  
+// Specify module with cloud mask functions.  
+var s2mask_tools = require(   'projects/gee-edu/book:Part F - Fundamentals/F6 - Advanced Topics/F6.2 Scaling Up/modules/s2cloudmask.js'  
+);

    Next, load and filter our three Sentinel-2 image collections.

    -

    // Sentinel-2 surface reflectance data for the composite.
    -var s2Sr = ee.ImageCollection(‘COPERNICUS/S2_SR’)
    -   .filterDate(start, end)
    -   .filterBounds(roi)
    -   .select([‘B2’, ‘B3’, ‘B4’, ‘B5’]);

    -

    // Sentinel-2 Level 1C data (top-of-atmosphere).  
    -// Bands B7, B8, B8A and B10 needed for CDI and the cloud mask function.
    -var s2 = ee.ImageCollection(‘COPERNICUS/S2’)
    -   .filterBounds(roi)
    -   .filterDate(start, end)
    -   .select([‘B7’, ‘B8’, ‘B8A’, ‘B10’]);

    -

    // Cloud probability dataset - used in cloud mask function
    -var s2c = ee.ImageCollection(‘COPERNICUS/S2_CLOUD_PROBABILITY’)
    -   .filterDate(start, end)
    -   .filterBounds(roi);

    +
    // Sentinel-2 surface reflectance data for the composite.  
+var s2Sr = ee.ImageCollection('COPERNICUS/S2_SR')  
+   .filterDate(start, end)  
+   .filterBounds(roi)  
+   .select(['B2', 'B3', 'B4', 'B5']);  
+  
+// Sentinel-2 Level 1C data (top-of-atmosphere).    
+// Bands B7, B8, B8A and B10 needed for CDI and the cloud mask function.  
+var s2 = ee.ImageCollection('COPERNICUS/S2')  
+   .filterBounds(roi)  
+   .filterDate(start, end)  
+   .select(['B7', 'B8', 'B8A', 'B10']);  
+  
+// Cloud probability dataset - used in cloud mask function  
+var s2c = ee.ImageCollection('COPERNICUS/S2_CLOUD_PROBABILITY')  
+   .filterDate(start, end)  
+   .filterBounds(roi);

    Now apply the cloud mask:

    -

    // Join the cloud probability dataset to surface reflectance.
    -var withCloudProbability = s2mask_tools.indexJoin(s2Sr, s2c,   ‘cloud_probability’);

    -

    // Join the L1C data to get the bands needed for CDI.
    -var withS2L1C = s2mask_tools.indexJoin(withCloudProbability, s2,   ‘l1c’);

    -

    // Map the cloud masking function over the joined collection.
    -// Cast output to ImageCollection
    -var masked = ee.ImageCollection(withS2L1C.map(s2mask_tools
    -.maskImage));

    +
    // Join the cloud probability dataset to surface reflectance.  
+var withCloudProbability = s2mask_tools.indexJoin(s2Sr, s2c,   'cloud_probability');  
+  
+// Join the L1C data to get the bands needed for CDI.  
+var withS2L1C = s2mask_tools.indexJoin(withCloudProbability, s2,   'l1c');  
+  
+// Map the cloud masking function over the joined collection.  
+// Cast output to ImageCollection  
+var masked = ee.ImageCollection(withS2L1C.map(s2mask_tools  
+.maskImage));

    Next, generate and visualize the median composite:

    -

    // Take the median, specifying a tileScale to avoid memory errors.
    -var median = masked.reduce(ee.Reducer.median(), 8);

    -

    // Display the results.
    -Map.centerObject(roi, 12);
    -Map.addLayer(roi);

    -

    var viz = {
    -   bands: [‘B4_median’, ‘B3_median’, ‘B2_median’],
    -   min: 0,
    -   max: 3000
    -};
    -Map.addLayer(median, viz, ‘median’);

    +
    // Take the median, specifying a tileScale to avoid memory errors.  
+var median = masked.reduce(ee.Reducer.median(), 8);  
+  
+// Display the results.  
+Map.centerObject(roi, 12);  
+Map.addLayer(roi);  
+  
+var viz = {  
+   bands: ['B4_median', 'B3_median', 'B2_median'],  
+   min: 0,  
+   max: 3000  
+};  
+Map.addLayer(median, viz, 'median');
    @@ -1561,75 +1793,118 @@ Note
    -

    Code Checkpoint F62c. The book’s repository contains a script that shows what your code should look like at this point.

    +

    Code Checkpoint F62c. The book’s repository contains a script that shows what your code should look like at this point.

    After about 1–3 minutes, Earth Engine returns our composite to us on the fly (Fig. F6.2.5). Note that panning and zooming to a new area requires that Earth Engine must again issue the compositing request to calculate the image for new areas. Given the delay, this isn’t a very satisfying way to explore our composite.

    -

    -

    Fig. F6.2.5 Map view of Seattle, Washington, USA (left) and the corresponding Sentinel-2 composite (right)

    +

    Fig. F6.2.5 Map view of Seattle, Washington, USA (left) and the corresponding Sentinel-2 composite (right)Fig. F6.2.5 Map view of Seattle, Washington, USA (left) and the corresponding Sentinel-2 composite (right)

    Next, expand our view (set zoom to 9) to exceed the limits of on-the-fly computation (Fig. F6.2.6).

    Map.centerObject(roi, 9);
    Map.addLayer(roi);
    Map.addLayer(median, viz, ‘median’);

    -

    -

    Fig. F6.2.6 Error message for exceeding memory limits in interactive mode

    +
    +
    +

    +

    Fig. F6.2.6 Error message for exceeding memory limits in interactive mode

    +
    +

    As you can see, this is an excellent candidate for an export task rather than running in “on-the-fly” interactive mode, as above.

    -
    -

    10.2.2 2.2. Generate a Regional Composite Through Spatial Tiling

    -

    Our goal is to apply the cloud masking method in Sect. 2.1 to the state of Washington, United States. In our testing, we successfully exported one Sentinel-2 composite for this area in about nine hours, but for this tutorial, let’s presume we need to split the task up to be successful.

    -

    Essentially, we want to split our region of interest up into a regular grid. For each grid, we will export a composite image into a new ImageCollection asset. We can then load and mosaic our composite for use in downstream scripts (see below).

    +
    +

    10.1.3 2.2. Generate a Regional Composite Through Spatial Tiling

    +

    Our goal is to apply the cloud masking method in Sect. 2.1 to the state of Washington, United States. In our testing, we successfully exported one Sentinel-2 composite for this area in about nine hours, but for this tutorial, let’s presume we need to split the task up to be successful.

    +

    Essentially, we want to split our region of interest up into a regular grid. For each grid, we will export a composite image into a new ImageCollection asset. We can then load and mosaic our composite for use in downstream scripts (see below).

    First, generate a spatial polygon grid (FeatureCollection) of desired size over your region of interest (see Fig. F6.2.7):

    -

    // Specify helper functions.
    -var s2mask_tools = require(   ‘projects/gee-edu/book:Part F - Fundamentals/F6 - Advanced Topics/F6.2 Scaling Up/modules/s2cloudmask.js’
    -);

    -

    // Set the Region of Interest: Washington, USA.
    -var roi = ee.FeatureCollection(‘TIGER/2018/States’)
    -   .filter(ee.Filter.equals(‘NAME’, ‘Washington’));

    -

    // Specify grid size in projection, x and y units (based on projection).
    -var projection = ‘EPSG:4326’; // WGS84 lat lon
    -var dx = 2.5;
    -var dy = 1.5;

    -

    // Dates over which to create a median composite.
    -var start = ee.Date(‘2019-03-01’);
    -var end = ee.Date(‘2019-09-01’);

    -

    // Make grid and visualize.
    -var proj = ee.Projection(projection).scale(dx, dy);
    -var grid = roi.geometry().coveringGrid(proj);

    -

    Map.addLayer(roi, {}, ‘roi’);
    -Map.addLayer(grid, {}, ‘grid’);

    -

    -

    Fig. F6.2.7 Visualization of the regular spatial grid generated for use in spatial batch processing

    -

    Next, create a new, empty ImageCollection asset to use as our export destination (Assets > New > Image Collection; Fig. F6.2.8). Name the image collection ‘S2_composite_WA’ and specify the asset location in your user folder (e.g., “path/to/your/asset/s2_composite_WA”).

    -

    -

    Fig. F6.2.8 The “create new image collection asset” menu in the Code Editor

    -

    Specify the ImageCollection to export to, along with a base name for each image (the tile number will be appended in the batch export).

    -

    // Export info.
    -var assetCollection = ‘path/to/your/asset/s2_composite_WA’;
    -var imageBaseName = ‘S2_median_’;

    -

    Extract grid numbers to use as looping variables. Note there is one getInfo call here, which should be used sparingly and never within a for-loop if you can help it. We use it to bring the number of grid cells we’ve generated onto the client-side to set up the for-loop over grids. Note that if your grid has too many elements, you may need a different strategy.

    -

    // Get a list based on grid number.
    -var gridSize = grid.size().getInfo();
    -var gridList = grid.toList(gridSize);

    +
    // Specify helper functions.  
+var s2mask_tools = require(   'projects/gee-edu/book:Part F - Fundamentals/F6 - Advanced Topics/F6.2 Scaling Up/modules/s2cloudmask.js'  
+);  
+  
+// Set the Region of Interest: Washington, USA.  
+var roi = ee.FeatureCollection('TIGER/2018/States')  
+   .filter(ee.Filter.equals('NAME', 'Washington'));  
+  
+// Specify grid size in projection, x and y units (based on projection).  
+var projection = 'EPSG:4326'; // WGS84 lat lon  
+var dx = 2.5;  
+var dy = 1.5;  
+  
+// Dates over which to create a median composite.  
+var start = ee.Date('2019-03-01');  
+var end = ee.Date('2019-09-01');  
+  
+// Make grid and visualize.  
+var proj = ee.Projection(projection).scale(dx, dy);  
+var grid = roi.geometry().coveringGrid(proj);  
+  
+Map.addLayer(roi, {}, 'roi');  
+Map.addLayer(grid, {}, 'grid');
    +
    +
    +

    +

    Fig. F6.2.7 Visualization of the regular spatial grid generated for use in spatial batch processing

    +
    +
    +

    Next, create a new, empty ImageCollection asset to use as our export destination (Assets > New > Image Collection; Fig. F6.2.8). Name the image collection ‘S2_composite_WA’ and specify the asset location in your user folder (e.g., “path/to/your/asset/s2_composite_WA”).

    +
    +
    +

    +

    Fig. F6.2.8 The “create new image collection asset” menu in the Code Editor

    +
    +
    +

    Specify the ImageCollection to export to, along with a base name for each image (the tile number will be appended in the batch export).

    +
    // Export info.  
+var assetCollection = 'path/to/your/asset/s2_composite_WA';  
+var imageBaseName = 'S2_median_';
    +

    Extract grid numbers to use as looping variables. Note there is one getInfo call here, which should be used sparingly and never within a for-loop if you can help it. We use it to bring the number of grid cells we’ve generated onto the client-side to set up the for-loop over grids. Note that if your grid has too many elements, you may need a different strategy.

    +
    // Get a list based on grid number.  
+var gridSize = grid.size().getInfo();  
+var gridList = grid.toList(gridSize);

    Batch generate a composite image task export for each grid via looping:

    -

    // In each grid cell, export a composite
    -for (var i = 0; i < gridSize; i++) {   // Extract grid polygon and filter S2 datasets for this region.   var gridCell = ee.Feature(gridList.get(i)).geometry();   var s2Sr = ee.ImageCollection(‘COPERNICUS/S2_SR’)
    -       .filterDate(start, end)
    -       .filterBounds(gridCell)
    -       .select([‘B2’, ‘B3’, ‘B4’, ‘B5’]);   var s2 = ee.ImageCollection(‘COPERNICUS/S2’)
    -       .filterDate(start, end)
    -       .filterBounds(gridCell)
    -       .select([‘B7’, ‘B8’, ‘B8A’, ‘B10’]);   var s2c = ee.ImageCollection(‘COPERNICUS/S2_CLOUD_PROBABILITY’)
    -       .filterDate(start, end)
    -       .filterBounds(gridCell);   // Apply the cloud mask.   var withCloudProbability = s2mask_tools.indexJoin(s2Sr, s2c,       ‘cloud_probability’);   var withS2L1C = s2mask_tools.indexJoin(withCloudProbability, s2,       ‘l1c’);   var masked = ee.ImageCollection(withS2L1C.map(s2mask_tools
    -       .maskImage));   // Generate a median composite and export.   var median = masked.reduce(ee.Reducer.median(), 8);   // Export.   var imagename = imageBaseName + ‘tile’ + i;   Export.image.toAsset({
    -       image: median,
    -       description: imagename,
    -       assetId: assetCollection + ‘/’ + imagename,
    -       scale: 10,
    -       region: gridCell,
    -       maxPixels: 1e13   });
    -}

    +
    // In each grid cell, export a composite  
+for (var i = 0; i < gridSize; i++) {   // Extract grid polygon and filter S2 datasets for this region.   var gridCell = ee.Feature(gridList.get(i)).geometry();   var s2Sr = ee.ImageCollection('COPERNICUS/S2_SR')  
+       .filterDate(start, end)  
+       .filterBounds(gridCell)  
+       .select(['B2', 'B3', 'B4', 'B5']);   var s2 = ee.ImageCollection('COPERNICUS/S2')  
+       .filterDate(start, end)  
+       .filterBounds(gridCell)  
+       .select(['B7', 'B8', 'B8A', 'B10']);   var s2c = ee.ImageCollection('COPERNICUS/S2_CLOUD_PROBABILITY')  
+       .filterDate(start, end)  
+       .filterBounds(gridCell);   // Apply the cloud mask.   var withCloudProbability = s2mask_tools.indexJoin(s2Sr, s2c,       'cloud_probability');   var withS2L1C = s2mask_tools.indexJoin(withCloudProbability, s2,       'l1c');   var masked = ee.ImageCollection(withS2L1C.map(s2mask_tools  
+       .maskImage));   // Generate a median composite and export.   var median = masked.reduce(ee.Reducer.median(), 8);   // Export.   var imagename = imageBaseName + 'tile' + i;   Export.image.toAsset({  
+       image: median,  
+       description: imagename,  
+       assetId: assetCollection + '/' + imagename,  
+       scale: 10,  
+       region: gridCell,  
+       maxPixels: 1e13   });  
+}
+
+:::{.callout-note}
+Code Checkpoint F62d. The book’s repository contains a script that shows what your code should look like at this point.
+:::
+Similar to Sect. 1.2, we now have a list of tasks to execute. We can hold the Cmd/Ctrl key and click Run to execute each task (Fig. F6.2.9). Again, users with applications requiring large batches may want to explore the Earth Engine Python API, which is well-suited to batching work. The output ImageCollection is 35.3 GB, so you may not want to execute all (or any) of these tasks but can access our pre-generated image, as discussed below.
+
+![Fig. F6.2.9 Spatial batch tasks have been generated and are ready to run](F6/image22.png)
+
+
+In addition to being necessary for very large regions, batch processing can speed things up for moderate scales. In our tests, tiles averaged about one hour to complete. Because three jobs in our queue were running simultaneously, we covered the full state of Washington in about four hours (compared to about nine hours when tested for the full state of Washington at once). Users should note, however, that there is also an overhead to spinning up each batch task. Finding the balance between task size and task number is a challenge for most Earth Engine users that becomes easier with experience.
+
+In a new script, load the exported ImageCollection and mosaic for use.
+
+// load image collection and mosaic into single image  
+var assetCollection = 'projects/gee-book/assets/F6-2/s2_composite_WA';  
+var composite = ee.ImageCollection(assetCollection).mosaic();  
+  
+// Display the results  
+var geometry = ee.Geometry.Point([-120.5873563817392,   47.39035206888694  
+]);  
+Map.centerObject(geometry, 6);  
+var vizParams = {  
+   bands: ['B4_median', 'B3_median', 'B2_median'],  
+   min: 0,  
+   max: 3000  
+};  
+Map.addLayer(composite, vizParams, 'median');
    @@ -1640,47 +1915,20 @@ Note
    -

    Code Checkpoint F62d. The book’s repository contains a script that shows what your code should look like at this point.

    +

    Code Checkpoint F62e. The book’s repository contains a script that shows what your code should look like at this point.

    -

    Similar to Sect. 1.2, we now have a list of tasks to execute. We can hold the Cmd/Ctrl key and click Run to execute each task (Fig. F6.2.9). Again, users with applications requiring large batches may want to explore the Earth Engine Python API, which is well-suited to batching work. The output ImageCollection is 35.3 GB, so you may not want to execute all (or any) of these tasks but can access our pre-generated image, as discussed below.

    -

    -

    Fig. F6.2.9 Spatial batch tasks have been generated and are ready to run

    -

    In addition to being necessary for very large regions, batch processing can speed things up for moderate scales. In our tests, tiles averaged about one hour to complete. Because three jobs in our queue were running simultaneously, we covered the full state of Washington in about four hours (compared to about nine hours when tested for the full state of Washington at once). Users should note, however, that there is also an overhead to spinning up each batch task. Finding the balance between task size and task number is a challenge for most Earth Engine users that becomes easier with experience.

    -

    In a new script, load the exported ImageCollection and mosaic for use.

    -

    // load image collection and mosaic into single image
    -var assetCollection = ‘projects/gee-book/assets/F6-2/s2_composite_WA’;
    -var composite = ee.ImageCollection(assetCollection).mosaic();

    -

    // Display the results
    -var geometry = ee.Geometry.Point([-120.5873563817392,   47.39035206888694
    -]);
    -Map.centerObject(geometry, 6);
    -var vizParams = {
    -   bands: [‘B4_median’, ‘B3_median’, ‘B2_median’],
    -   min: 0,
    -   max: 3000
    -};
    -Map.addLayer(composite, vizParams, ‘median’);

    -
    -
    -
    - +
    +
    +

    +

    Fig. F6.2.10 Sentinel-2 composite covering the state of Washington, loaded from asset. The remaining white colors are snow-capped mountains, not clouds.

    +
    -
    -Note -
    -
    -
    -

    Code Checkpoint F62e. The book’s repository contains a script that shows what your code should look like at this point.

    -
    -
    -

    -

    Fig. F6.2.10 Sentinel-2 composite covering the state of Washington, loaded from asset. The remaining white colors are snow-capped mountains, not clouds.

    Note the ease, speed, and joy of panning and zooming to explore the pre-computed composite asset (Fig. F6.2.10) compared to the on-the-fly version discussed in Sect. 2.1.

    -
    -

    10.3 Multistep Workflows and Intermediate Assets

    +
    +

    10.2 Multistep Workflows and Intermediate Assets

    Often, our goals require several processing steps that cannot be completed within one Earth Engine computational chain. In these cases, the best strategy becomes breaking down tasks into individual pieces that are created, stored in assets, and used across several scripts. Each sequential script creates an intermediate output, and this intermediate output becomes the input to the next script.

    As an example, consider the land use classification task of identifying irrigated agricultural lands. This type of classification can benefit from several types of evidence, including satellite composites, aggregated weather information, soil information, and/or crop type locations. Individual steps for this type of work might include:

      @@ -1695,27 +1943,39 @@ Note

      Multipart workflows can become unwieldy to manage, particularly if there are multiple collaborators or the project has a long timeline; it can be difficult to remember why each script was developed and where it fits in the overall workflow.

      Here, we provide tips for managing multipart workflows. These are somewhat opinionated and based largely on concepts from “Good Enough Practices in Scientific Computing” (Wilson et al. 2017). Ultimately, your personal workflow practices will be a combination of what works for you, what works for your larger team and organization, and, hopefully, what works for good documentation and reproducibility.

      Tip 1. Create a repository for each project

      -

      The repository can be considered the fundamental project unit. In Earth Engine, sharing permissions are set for each individual repository, so this allows you to share a specific project with others (see Chap. F6.1).

      -

      By default, Earth Engine saves new scripts in a “default” repository specific for each user (users//default). You can create new repositories on the Scripts tab of the Code Editor (Fig. F6.2.11).

      -

      -

      Fig. F6.2.11 The Code Editor menu for creating new repositories

      +

      The repository can be considered the fundamental project unit. In Earth Engine, sharing permissions are set for each individual repository, so this allows you to share a specific project with others (see Chap. F6.1).

      +

      By default, Earth Engine saves new scripts in a “default” repository specific for each user (users//default). You can create new repositories on the Scripts tab of the Code Editor (Fig. F6.2.11).

      +
      +
      +

      +

      Fig. F6.2.11 The Code Editor menu for creating new repositories

      +
      +

      To adjust permissions for each repository, click on the Gear icon (Fig. F6.2.12):

      -

      -

      Fig. F6.2.12 Access the sharing and permissions menu for each repository by clicking the Gear icon

      +
      +
      +

      +

      Fig. F6.2.12 Access the sharing and permissions menu for each repository by clicking the Gear icon

      +
      +

      For users familiar with version control, Earth Engine uses a git-based script manager, so each repository can also be managed, edited, and/or synced with your local copy or collaborative spaces like GitHub.

      Tip 2. Make a separate script for each step, and make script file names informative and self-sorting

      Descriptive, self-sorting filenames are an excellent “good enough” way to keep your projects organized. We recommend starting script names with zero-padded numeric values to take advantage of default ordering. Because we are generating assets in early scripts that are used in later scripts, it’s important to preserve the order of your workflow. The name should also include short descriptions of what the script does (Fig. F6.2.13).

      -

      -

      Fig. F6.2.13 An example project repository with multiple scripts. Using leading numbers when naming scripts allows you to order them by their position in the workflow.

      +
      +
      +

      +

      Fig. F6.2.13 An example project repository with multiple scripts. Using leading numbers when naming scripts allows you to order them by their position in the workflow.

      +
      +

      Leaving some decimal places between successive scripts gives you the ability to easily insert any additional steps you didn’t originally anticipate. And zero-padding means your self-sorting still works once you move into double-digit numbers.

      Other script organization strategies might involve including subfolders to collect scripts related to main steps. For example, one could have a subfolder “04_classifiers” to keep alternative classification scripts in one place, using a more tree-based file structure. Again, each user or group will develop a system that works for them. The important part is to have an organizational system.

      Tip 3. Consider data types and file sizes when storing intermediates

      Images and image collections are common intermediate file types, since generating satellite composites or creating land use classifications tends to be computationally intensive. These assets can also be quite large, depending on the resolution and region size. Recall that our single-year, subregional Sentinel-2 composite in Sect. 2 was about 23 GB.

      -

      Image values can be stored from 8-bit integers to 64-bit double floats (numbers with decimals). Higher bits allow for more precision, but have much larger file sizes and are not always necessary. For example, if we are generating a land use map with five classes, we can convert that to a signed or unsigned 8-bit integer using toInt8 or toUint8 prior to exporting to asset, which can accommodate 256 unique values. This results in a smaller file size. Selectively retaining only bands of interest is also helpful to reduce size.

      -

      For cases requiring decimals and precision, consider whether a 32-bit float will do the job instead of a 64-bit double—toFloat will convert an image to a 32-bit float. If you find you need to conserve storage, you can also scale float values and store as an integer image (image.multiply(100).toInt16(), for example). This would retain precision to the second decimal place and reduce file size by a factor of two. Note that this may require you to unscale the values in downstream use. Ultimately, the appropriate data type will be specific to your needs.

      +

      Image values can be stored from 8-bit integers to 64-bit double floats (numbers with decimals). Higher bits allow for more precision, but have much larger file sizes and are not always necessary. For example, if we are generating a land use map with five classes, we can convert that to a signed or unsigned 8-bit integer using toInt8 or toUint8 prior to exporting to asset, which can accommodate 256 unique values. This results in a smaller file size. Selectively retaining only bands of interest is also helpful to reduce size.

      +

      For cases requiring decimals and precision, consider whether a 32-bit float will do the job instead of a 64-bit double—toFloat will convert an image to a 32-bit float. If you find you need to conserve storage, you can also scale float values and store as an integer image (image.multiply(100).toInt16(), for example). This would retain precision to the second decimal place and reduce file size by a factor of two. Note that this may require you to unscale the values in downstream use. Ultimately, the appropriate data type will be specific to your needs.

      And of course, as mentioned above under “The Importance of Best Coding Practices,” be aware of the scale resolution you are working at, and avoid using unnecessarily high resolution when it’s not supported by either the input imagery or your research goals.

      Tip 4. Consider Google Cloud Platform for hosting larger intermediates

      -

      If you are working with very large or very many files, you can link Earth Engine with Cloud Projects on Google Cloud Platform. See the Earth Engine documentation on “Setting Up Earth Engine Enabled Cloud Projects” for more information.

      +

      If you are working with very large or very many files, you can link Earth Engine with Cloud Projects on Google Cloud Platform. See the Earth Engine documentation on “Setting Up Earth Engine Enabled Cloud Projects” for more information.

    Synthesis

    @@ -1743,21 +2003,21 @@ Chapter Information

    Author

    -

    Qiusheng Wu

    +

    Qiusheng Wu

    Overview

    -

    The purpose of this chapter is to demonstrate how to design and publish Earth Engine Apps using both JavaScript and Python. You will be introduced to the Earth Engine User Interface JavaScript API and the geemap Python package. Upon completion of this chapter, you will be able to publish an Earth Engine App with a split-panel map for visualizing land cover change.

    +

    The purpose of this chapter is to demonstrate how to design and publish Earth Engine Apps using both JavaScript and Python. You will be introduced to the Earth Engine User Interface JavaScript API and the geemap Python package. Upon completion of this chapter, you will be able to publish an Earth Engine App with a split-panel map for visualizing land cover change.

    Learning Outcomes

      -
    • Designing a user interface for an Earth Engine App using JavaScript.
      • +
    • Designing a user interface for an Earth Engine App using JavaScript.
    • Publishing an Earth Engine App for visualizing land cover change.
    • Developing an Earth Engine App using Python and geemap.
    • Deploying an Earth Engine App using a local computer as a web server.
    • Publishing an Earth Engine App using Python and free cloud platforms.
      • -
    • Creating a conda environment using Anaconda/Miniconda.
      • +
    • Creating a conda environment using Anaconda/Miniconda.
    • Installing Python packages and using Jupyter Notebook.
    • Commiting changes to a GitHub repository.
    @@ -1773,115 +2033,131 @@ Chapter Information

    Introduction

    -

    Earth Engine has a user interface API that allows users to build and publish interactive web apps directly from the JavaScript Code Editor. Many readers will have encountered a call to ui.Chart in other chapters, but much more interface functionality is available. In particular, users can utilize the ui functions to construct an entire graphical user interface (GUI) for their Earth Engine script. The GUI may include simple widgets (e.g., labels, buttons, checkboxes, sliders, text boxes) as well as more complex widgets (e.g., charts, maps, panels) for controlling the GUI layout. A complete list of the ui widgets and more information about panels can be found at the links below. Once a GUI is constructed, users can publish the App from the JavaScript Code Editor by clicking the Apps button above the script panel in the Code Editor.

    +

    Earth Engine has a user interface API that allows users to build and publish interactive web apps directly from the JavaScript Code Editor. Many readers will have encountered a call to ui.Chart in other chapters, but much more interface functionality is available. In particular, users can utilize the ui functions to construct an entire graphical user interface (GUI) for their Earth Engine script. The GUI may include simple widgets (e.g., labels, buttons, checkboxes, sliders, text boxes) as well as more complex widgets (e.g., charts, maps, panels) for controlling the GUI layout. A complete list of the ui widgets and more information about panels can be found at the links below. Once a GUI is constructed, users can publish the App from the JavaScript Code Editor by clicking the Apps button above the script panel in the Code Editor.

    -

    Unlike the Earth Engine JavaScript API, the Earth Engine Python API does not provide functionality for building interactive user interfaces. Fortunately, the Jupyter ecosystem has ipywidgets, an architecture for creating interactive user interface controls (e.g., buttons, sliders, checkboxes, text boxes, dropdown lists) in Jupyter notebooks that communicate with Python code. The integration of graphical widgets into the Jupyter Notebook workflow allows users to configure ad hoc control panels to interactively sweep over parameters using graphical widget controls. One very powerful widget is the output widget, which can be used to display rich output generated by IPython, such as text, images, charts, and videos. A complete list of widgets and more information about the output widget can be found at the links below. By integrating ipyleaflet (for creating interactive maps) and ipywidgets (for designing interactive user interfaces), the geemap Python package (https://geemap.org) makes it much easier to explore and analyze massive Earth Engine datasets via a web browser in a Jupyter environment suitable for interactive exploration, teaching, and sharing. Users can build interactive Earth Engine Apps using geemap with minimal coding (Fig. F6.3.1).

    +

    Unlike the Earth Engine JavaScript API, the Earth Engine Python API does not provide functionality for building interactive user interfaces. Fortunately, the Jupyter ecosystem has ipywidgets, an architecture for creating interactive user interface controls (e.g., buttons, sliders, checkboxes, text boxes, dropdown lists) in Jupyter notebooks that communicate with Python code. The integration of graphical widgets into the Jupyter Notebook workflow allows users to configure ad hoc control panels to interactively sweep over parameters using graphical widget controls. One very powerful widget is the output widget, which can be used to display rich output generated by IPython, such as text, images, charts, and videos. A complete list of widgets and more information about the output widget can be found at the links below. By integrating ipyleaflet (for creating interactive maps) and ipywidgets (for designing interactive user interfaces), the geemap Python package (https://geemap.org) makes it much easier to explore and analyze massive Earth Engine datasets via a web browser in a Jupyter environment suitable for interactive exploration, teaching, and sharing. Users can build interactive Earth Engine Apps using geemap with minimal coding (Fig. F6.3.1).

    -

    -

    Fig. F6.3.1 The GUI of geemap in a Jupyter environment

    +
    +
    +

    +

    Fig. F6.3.1 The GUI of geemap in a Jupyter environment

    +
    +

    11.1 Building an Earth Engine App Using JavaScript

    -

    In this section, you will learn how to design a user interface for an Earth Engine App using JavaScript and the Earth Engine User Interface API. Upon completion of this section, you will have an Earth Engine App with a split-panel map for visualizing land cover change using the Landsat-based United States Geological Survey National Land Cover Database (NLCD).

    -

    First, let’s define a function for filtering the NLCD ImageCollection by year and select the landcover band. The function returns an Earth Engine ui.Map.Layer of the landcover band of the selected NLCD image. Note that as of this writing, NLCD spans nine epochs: 1992, 2001, 2004, 2006, 2008, 2011, 2013, 2016, and 2019. The 1992 data are primarily based on unsupervised classification of Landsat data, while the rest of the images rely on the imperviousness data layer for the urban classes and on a decision-tree classification for the rest. The 1992 image is not directly comparable to any later editions of NLCD (see the Earth Engine Data Catalog for more details, if needed). Therefore, we will use only the eight epochs after 2000 in this lab.  

    -

    // Get an NLCD image by year.
    -var getNLCD = function(year) {   // Import the NLCD collection.   var dataset = ee.ImageCollection(       ‘USGS/NLCD_RELEASES/2019_REL/NLCD’);   // Filter the collection by year.   var nlcd = dataset.filter(ee.Filter.eq(‘system:index’, year))
    -       .first();   // Select the land cover band.   var landcover = nlcd.select(‘landcover’);   return ui.Map.Layer(landcover, {}, year);
    -};

    -

    Our intention is to create a dropdown list so that when a particular epoch is selected, the corresponding NLCD image layer will be displayed on the map. We’ll define a dictionary with each NLCD epoch as the key and its corresponding NLCD image layer as the value. The keys of the dictionary (i.e., the eight NLCD epochs) will be used as the input to the dropdown lists (ui.Select) on the split-level map.

    -

    // Create a dictionary with each year as the key
    -// and its corresponding NLCD image layer as the value.
    -var images = {   ‘2001’: getNLCD(‘2001’),   ‘2004’: getNLCD(‘2004’),   ‘2006’: getNLCD(‘2006’),   ‘2008’: getNLCD(‘2008’),   ‘2011’: getNLCD(‘2011’),   ‘2013’: getNLCD(‘2013’),   ‘2016’: getNLCD(‘2016’),   ‘2019’: getNLCD(‘2019’),
    -};

    -

    The split-panel map is composed of two individual maps, leftMap and rightMap. The map controls (e.g., zoomControl, scaleControl, mapTypeControl) will be shown only on rightMap. A control panel (ui.Panel) composed of a label (ui.Label) and a dropdown list (ui.Select) is added to each map. When an NLCD epoch is selected from a dropdown list, the function updateMap will be called to show the corresponding image layer of the selected epoch.

    -

    // Create the left map, and have it display the first layer.
    -var leftMap = ui.Map();
    -leftMap.setControlVisibility(false);
    -var leftSelector = addLayerSelector(leftMap, 0, ‘top-left’);

    -

    // Create the right map, and have it display the last layer.
    -var rightMap = ui.Map();
    -rightMap.setControlVisibility(true);
    -var rightSelector = addLayerSelector(rightMap, 7, ‘top-right’);

    -

    // Adds a layer selection widget to the given map, to allow users to
    -// change which image is displayed in the associated map.
    -function addLayerSelector(mapToChange, defaultValue, position) {   var label = ui.Label(‘Select a year:’);   // This function changes the given map to show the selected image.   function updateMap(selection) {
    -       mapToChange.layers().set(0, images[selection]);
    -   }   // Configure a selection dropdown to allow the user to choose   // between images, and set the map to update when a user   // makes a selection.   var select = ui.Select({
    -       items: Object.keys(images),
    -       onChange: updateMap
    -   });
    -   select.setValue(Object.keys(images)[defaultValue], true);   var controlPanel =       ui.Panel({
    -           widgets: [label, select],
    -           style: {
    -               position: position
    -           }
    -       });

    -

       mapToChange.add(controlPanel);
    -}

    -

    When displaying a land cover classification image on the Map, it would be useful to add a legend to make it easier for users to interpret the land cover type associated with each color. Let’s define a dictionary that will be used to construct the legend. The dictionary contains two keys: names (a list of land cover types) and colors (a list of colors associated with each land cover type). The legend will be placed in the bottom right of the Map.    

    -

    // Set the legend title.
    -var title = ‘NLCD Land Cover Classification’;

    -

    // Set the legend position.
    -var position = ‘bottom-right’;

    -

    // Define a dictionary that will be used to make a legend
    -var dict = {   ‘names’: [       ‘11        Open Water’,       ‘12        Perennial Ice/Snow’,       ‘21        Developed, Open Space’,       ‘22        Developed, Low Intensity’,       ‘23        Developed, Medium Intensity’,       ‘24        Developed, High Intensity’,       ‘31        Barren Land (Rock/Sand/Clay)’,       ‘41        Deciduous Forest’,       ‘42        Evergreen Forest’,       ‘43        Mixed Forest’,       ‘51        Dwarf Scrub’,       ‘52        Shrub/Scrub’,       ‘71        Grassland/Herbaceous’,       ‘72        Sedge/Herbaceous’,       ‘73        Lichens’,       ‘74        Moss’,       ‘81        Pasture/Hay’,       ‘82        Cultivated Crops’,       ‘90        Woody Wetlands’,       ‘95        Emergent Herbaceous Wetlands’,
    -   ],   ‘colors’: [       ‘#466b9f’, ‘#d1def8’, ‘#dec5c5’, ‘#d99282’, ‘#eb0000’,       ‘#ab0000’,       ‘#b3ac9f’, ‘#68ab5f’, ‘#1c5f2c’, ‘#b5c58f’, ‘#af963c’,       ‘#ccb879’,       ‘#dfdfc2’, ‘#d1d182’, ‘#a3cc51’, ‘#82ba9e’, ‘#dcd939’,       ‘#ab6c28’,       ‘#b8d9eb’, ‘#6c9fb8’,
    -   ]
    -};

    +

    In this section, you will learn how to design a user interface for an Earth Engine App using JavaScript and the Earth Engine User Interface API. Upon completion of this section, you will have an Earth Engine App with a split-panel map for visualizing land cover change using the Landsat-based United States Geological Survey National Land Cover Database (NLCD).

    +

    First, let’s define a function for filtering the NLCD ImageCollection by year and select the landcover band. The function returns an Earth Engine ui.Map.Layer of the landcover band of the selected NLCD image. Note that as of this writing, NLCD spans nine epochs: 1992, 2001, 2004, 2006, 2008, 2011, 2013, 2016, and 2019. The 1992 data are primarily based on unsupervised classification of Landsat data, while the rest of the images rely on the imperviousness data layer for the urban classes and on a decision-tree classification for the rest. The 1992 image is not directly comparable to any later editions of NLCD (see the Earth Engine Data Catalog for more details, if needed). Therefore, we will use only the eight epochs after 2000 in this lab.

    +
    // Get an NLCD image by year.  
+var getNLCD = function(year) {   // Import the NLCD collection.   var dataset = ee.ImageCollection(       'USGS/NLCD_RELEASES/2019_REL/NLCD');   // Filter the collection by year.   var nlcd = dataset.filter(ee.Filter.eq('system:index', year))  
+       .first();   // Select the land cover band.   var landcover = nlcd.select('landcover');   return ui.Map.Layer(landcover, {}, year);  
+};
    +

    Our intention is to create a dropdown list so that when a particular epoch is selected, the corresponding NLCD image layer will be displayed on the map. We’ll define a dictionary with each NLCD epoch as the key and its corresponding NLCD image layer as the value. The keys of the dictionary (i.e., the eight NLCD epochs) will be used as the input to the dropdown lists (ui.Select) on the split-level map.

    +
    // Create a dictionary with each year as the key  
+// and its corresponding NLCD image layer as the value.  
+var images = {   '2001': getNLCD('2001'),   '2004': getNLCD('2004'),   '2006': getNLCD('2006'),   '2008': getNLCD('2008'),   '2011': getNLCD('2011'),   '2013': getNLCD('2013'),   '2016': getNLCD('2016'),   '2019': getNLCD('2019'),  
+};
    +

    The split-panel map is composed of two individual maps, leftMap and rightMap. The map controls (e.g., zoomControl, scaleControl, mapTypeControl) will be shown only on rightMap. A control panel (ui.Panel) composed of a label (ui.Label) and a dropdown list (ui.Select) is added to each map. When an NLCD epoch is selected from a dropdown list, the function updateMap will be called to show the corresponding image layer of the selected epoch.

    +
    // Create the left map, and have it display the first layer.  
+var leftMap = ui.Map();  
+leftMap.setControlVisibility(false);  
+var leftSelector = addLayerSelector(leftMap, 0, 'top-left');  
+  
+// Create the right map, and have it display the last layer.  
+var rightMap = ui.Map();  
+rightMap.setControlVisibility(true);  
+var rightSelector = addLayerSelector(rightMap, 7, 'top-right');  
+  
+// Adds a layer selection widget to the given map, to allow users to  
+// change which image is displayed in the associated map.  
+function addLayerSelector(mapToChange, defaultValue, position) {   var label = ui.Label('Select a year:');   // This function changes the given map to show the selected image.   function updateMap(selection) {  
+       mapToChange.layers().set(0, images[selection]);  
+   }   // Configure a selection dropdown to allow the user to choose   // between images, and set the map to update when a user   // makes a selection.   var select = ui.Select({  
+       items: Object.keys(images),  
+       onChange: updateMap  
+   });  
+   select.setValue(Object.keys(images)[defaultValue], true);   var controlPanel =       ui.Panel({  
+           widgets: [label, select],  
+           style: {  
+               position: position  
+           }  
+       });  
+  
+   mapToChange.add(controlPanel);  
+}
+
+When displaying a land cover classification image on the Map, it would be useful to add a legend to make it easier for users to interpret the land cover type associated with each color. Let’s define a dictionary that will be used to construct the legend. The dictionary contains two keys: names (a list of land cover types) and colors (a list of colors associated with each land cover type). The legend will be placed in the bottom right of the Map.    
+
+// Set the legend title.  
+var title = 'NLCD Land Cover Classification';  
+  
+// Set the legend position.  
+var position = 'bottom-right';  
+  
+// Define a dictionary that will be used to make a legend  
+var dict = {   'names': [       '11        Open Water',       '12        Perennial Ice/Snow',       '21        Developed, Open Space',       '22        Developed, Low Intensity',       '23        Developed, Medium Intensity',       '24        Developed, High Intensity',       '31        Barren Land (Rock/Sand/Clay)',       '41        Deciduous Forest',       '42        Evergreen Forest',       '43        Mixed Forest',       '51        Dwarf Scrub',       '52        Shrub/Scrub',       '71        Grassland/Herbaceous',       '72        Sedge/Herbaceous',       '73        Lichens',       '74        Moss',       '81        Pasture/Hay',       '82        Cultivated Crops',       '90        Woody Wetlands',       '95        Emergent Herbaceous Wetlands',  
+   ],   'colors': [       '#466b9f', '#d1def8', '#dec5c5', '#d99282', '#eb0000',       '#ab0000',       '#b3ac9f', '#68ab5f', '#1c5f2c', '#b5c58f', '#af963c',       '#ccb879',       '#dfdfc2', '#d1d182', '#a3cc51', '#82ba9e', '#dcd939',       '#ab6c28',       '#b8d9eb', '#6c9fb8',  
+   ]  
+};

    With the legend dictionary defined above, we can now create a panel to hold the legend widget and add it to the Map. Each row on the legend widget is composed of a color box followed by its corresponding land cover type.

    -

    // Create a panel to hold the legend widget.
    -var legend = ui.Panel({
    -   style: {
    -       position: position,
    -       padding: ‘8px 15px’   }
    -});// Function to generate the legend.
    -function addCategoricalLegend(panel, dict, title) {   // Create and add the legend title.   var legendTitle = ui.Label({
    -       value: title,
    -       style: {
    -           fontWeight: ‘bold’,
    -           fontSize: ‘18px’,
    -           margin: ‘0 0 4px 0’,
    -           padding: ‘0’       }
    -   });
    -   panel.add(legendTitle);   var loading = ui.Label(‘Loading legend…’, {
    -       margin: ‘2px 0 4px 0’   });
    -   panel.add(loading);   // Creates and styles 1 row of the legend.   var makeRow = function(color, name) {       // Create the label that is actually the colored box.       var colorBox = ui.Label({
    -           style: {
    -               backgroundColor: color,               // Use padding to give the box height and width.               padding: ‘8px’,
    -               margin: ‘0 0 4px 0’           }
    -       });       // Create the label filled with the description text.       var description = ui.Label({
    -           value: name,
    -           style: {
    -               margin: ‘0 0 4px 6px’           }
    -       });       return ui.Panel({
    -           widgets: [colorBox, description],
    -           layout: ui.Panel.Layout.Flow(‘horizontal’)
    -       });
    -   };   // Get the list of palette colors and class names from the image.   var palette = dict.colors;   var names = dict.names;
    -   loading.style().set(‘shown’, false);   for (var i = 0; i < names.length; i++) {
    -       panel.add(makeRow(palette[i], names[i]));
    -   }

    -

       rightMap.add(panel);

    -

    }

    -

    The last step is to create a split-panel map to hold the linked maps (leftMap and rightMap) and tie everything together. When users pan and zoom one map, the other map will also be panned and zoomed to the same extent automatically. When users select a year from a dropdown list, the image layer will be updated accordingly. Users can use the slider to swipe through and visualize land cover change easily (Fig. F6.3.2). Please make sure you minimize the Code Editor and maximize the Map so that you can see the dropdown widget in the upper-right corner of the map.

    -

    addCategoricalLegend(legend, dict, title);

    -

    // Create a SplitPanel to hold the adjacent, linked maps.
    -var splitPanel = ui.SplitPanel({
    -   firstPanel: leftMap,
    -   secondPanel: rightMap,
    -   wipe: true,
    -   style: {
    -       stretch: ‘both’   }
    -});// Set the SplitPanel as the only thing in the UI root.
    -ui.root.widgets().reset([splitPanel]);
    -var linker = ui.Map.Linker([leftMap, rightMap]);
    -leftMap.setCenter(-100, 40, 4);

    +
    // Create a panel to hold the legend widget.  
+var legend = ui.Panel({  
+   style: {  
+       position: position,  
+       padding: '8px 15px'   }  
+});// Function to generate the legend.  
+function addCategoricalLegend(panel, dict, title) {   // Create and add the legend title.   var legendTitle = ui.Label({  
+       value: title,  
+       style: {  
+           fontWeight: 'bold',  
+           fontSize: '18px',  
+           margin: '0 0 4px 0',  
+           padding: '0'       }  
+   });  
+   panel.add(legendTitle);   var loading = ui.Label('Loading legend...', {  
+       margin: '2px 0 4px 0'   });  
+   panel.add(loading);   // Creates and styles 1 row of the legend.   var makeRow = function(color, name) {       // Create the label that is actually the colored box.       var colorBox = ui.Label({  
+           style: {  
+               backgroundColor: color,               // Use padding to give the box height and width.               padding: '8px',  
+               margin: '0 0 4px 0'           }  
+       });       // Create the label filled with the description text.       var description = ui.Label({  
+           value: name,  
+           style: {  
+               margin: '0 0 4px 6px'           }  
+       });       return ui.Panel({  
+           widgets: [colorBox, description],  
+           layout: ui.Panel.Layout.Flow('horizontal')  
+       });  
+   };   // Get the list of palette colors and class names from the image.   var palette = dict.colors;   var names = dict.names;  
+   loading.style().set('shown', false);   for (var i = 0; i < names.length; i++) {  
+       panel.add(makeRow(palette[i], names[i]));  
+   }  
+  
+   rightMap.add(panel);  
+  
+}
+
+The last step is to create a split-panel map to hold the linked maps (leftMap and rightMap) and tie everything together. When users pan and zoom one map, the other map will also be panned and zoomed to the same extent automatically. When users select a year from a dropdown list, the image layer will be updated accordingly. Users can use the slider to swipe through and visualize land cover change easily (Fig. F6.3.2). Please make sure you minimize the Code Editor and maximize the Map so that you can see the dropdown widget in the upper-right corner of the map.
+
+addCategoricalLegend(legend, dict, title);  
+  
+// Create a SplitPanel to hold the adjacent, linked maps.  
+var splitPanel = ui.SplitPanel({  
+   firstPanel: leftMap,  
+   secondPanel: rightMap,  
+   wipe: true,  
+   style: {  
+       stretch: 'both'   }  
+});// Set the SplitPanel as the only thing in the UI root.  
+ui.root.widgets().reset([splitPanel]);  
+var linker = ui.Map.Linker([leftMap, rightMap]);  
+leftMap.setCenter(-100, 40, 4);
    @@ -1892,30 +2168,50 @@ Note
    -

    Code Checkpoint F63a. The book’s repository contains a script that shows what your code should look like at this point.

    +

    Code Checkpoint F63a. The book’s repository contains a script that shows what your code should look like at this point.

    -

    -

    Fig. F6.3.2 A split-panel map for visualizing land cover change using NLCD

    +
    +
    +

    +

    Fig. F6.3.2 A split-panel map for visualizing land cover change using NLCD

    +
    +

    11.2 Publishing an Earth Engine App from the Code Editor

    -

    The goal of this section is to publish the Earth Engine App that we created in Sect. 1. The look and feel of interfaces changes often; if the exact windows described below change over time, the concepts should remain stable to help you to publish your App. First, load the script (see the Code Checkpoint in Sect. 1) into the Code Editor. Then, open the Manage Apps panel by clicking the Apps button above the script panel in the Code Editor (Fig. F6.3.3).

    -

    -

    Fig. F6.3.3 The Apps button in the JavaScript Code Editor

    -

    Now click on the New App button (Fig. F6.3.4).

    -

    -

    Fig. F6.3.4 The New App button

    -

    In the Publish New App dialog (Fig. F6.3.5), choose a name for the App (e.g., NLCD Land Cover Change), select a Google Cloud Project, provide a thumbnail to be shown in the Public Apps Gallery, and specify the location of the App’s source code. You may restrict access to the App to a particular Google Group or make it publicly accessible. Check Feature this app in your Public Apps Gallery if you would like this App to appear in your public gallery of Apps available at https://USERNAME.users.earthengine.app. When all fields are filled out and validated, the Publish button will be enabled; click it to complete publishing the App. 

    -

    -

    Fig. F6.3.5 The Publish New App dialog

    -

    To manage an App from the Code Editor, open the Manage Apps panel (Fig. F6.3.6) by clicking the Apps button above the script panel in the Code Editor (Fig. F6.3.3). There, you can update your App’s configuration or delete the App.

    -

    -

    Fig. F6.3.6 The Manage Apps panel

    +

    The goal of this section is to publish the Earth Engine App that we created in Sect. 1. The look and feel of interfaces changes often; if the exact windows described below change over time, the concepts should remain stable to help you to publish your App. First, load the script (see the Code Checkpoint in Sect. 1) into the Code Editor. Then, open the Manage Apps panel by clicking the Apps button above the script panel in the Code Editor (Fig. F6.3.3).

    +
    +
    +

    +

    Fig. F6.3.3 The Apps button in the JavaScript Code Editor

    +
    +
    +

    Now click on the New App button (Fig. F6.3.4).

    +
    +
    +

    +

    Fig. F6.3.4 The New App button

    +
    +
    +

    In the Publish New App dialog (Fig. F6.3.5), choose a name for the App (e.g., NLCD Land Cover Change), select a Google Cloud Project, provide a thumbnail to be shown in the Public Apps Gallery, and specify the location of the App’s source code. You may restrict access to the App to a particular Google Group or make it publicly accessible. Check Feature this app in your Public Apps Gallery if you would like this App to appear in your public gallery of Apps available at https://USERNAME.users.earthengine.app. When all fields are filled out and validated, the Publish button will be enabled; click it to complete publishing the App.

    +
    +
    +

    +

    Fig. F6.3.5 The Publish New App dialog

    +
    +
    +

    To manage an App from the Code Editor, open the Manage Apps panel (Fig. F6.3.6) by clicking the Apps button above the script panel in the Code Editor (Fig. F6.3.3). There, you can update your App’s configuration or delete the App.

    +
    +
    +

    +

    Fig. F6.3.6 The Manage Apps panel

    +
    +

    11.3 Developing an Earth Engine App Using geemap

    -

    In this section, you will learn how to develop an Earth Engine App using the geemap Python package and Jupyter Notebook. The geemap package is available on both PyPI (pip) and conda-forge. It is highly recommended that you create a fresh conda environment to install geemap.

    +

    In this section, you will learn how to develop an Earth Engine App using the geemap Python package and Jupyter Notebook. The geemap package is available on both PyPI (pip) and conda-forge. It is highly recommended that you create a fresh conda environment to install geemap.

    @@ -1926,17 +2222,17 @@ Note
    -

    Code Checkpoint F63b. The book’s repository contains information about setting up a conda environment and installing geemap.

    +

    Code Checkpoint F63b. The book’s repository contains information about setting up a conda environment and installing geemap.

    Once you have launched a Jupyter notebook in your browser, you can continue with the next steps of the lab.

    -

    On the Jupyter Notebook interface, click the New button in the upper-right corner and select Python 3 to create a new notebook. Change the name of the notebook from “Untitled” to something meaningful (e.g., “nlcd_app”). With the newly created notebook, we can start writing and executing Python code.

    -

    First, let’s import the Earth Engine and geemap libraries. Press Alt + Enter to execute the code and create a new cell below.

    -

    import eeimport geemap

    -

    Create an interactive map by specifying the map center (latitude, longitude) and zoom level (1–18). If this is the first time you use geemap and the Earth Engine Python API, a new tab will open in your browser asking you to authenticate Earth Engine. Follow the on-screen instructions to authenticate Earth Engine.

    +

    On the Jupyter Notebook interface, click the New button in the upper-right corner and select Python 3 to create a new notebook. Change the name of the notebook from “Untitled” to something meaningful (e.g., “nlcd_app”). With the newly created notebook, we can start writing and executing Python code.

    +

    First, let’s import the Earth Engine and geemap libraries. Press Alt + Enter to execute the code and create a new cell below.

    +

    import eeimport geemap

    +

    Create an interactive map by specifying the map center (latitude, longitude) and zoom level (1–18). If this is the first time you use geemap and the Earth Engine Python API, a new tab will open in your browser asking you to authenticate Earth Engine. Follow the on-screen instructions to authenticate Earth Engine.

    Map = geemap.Map(center=[40, -100], zoom=4)
    Map

    -

    Retrieve the NLCD 2019 image by filtering the NLCD ImageCollection and selecting the landcover band. Display the NLCD 2019 image on the interactive Map by using Map.addLayer.

    +

    Retrieve the NLCD 2019 image by filtering the NLCD ImageCollection and selecting the landcover band. Display the NLCD 2019 image on the interactive Map by using Map.addLayer.

    @@ -1954,17 +2250,21 @@ Map

    15 Display land cover on the map.Map.addLayer(landcover, {}, ‘NLCD 2019’)

    Map

    -

    Next, add the NLCD legend to the Map. The geemap package has several built-in legends, including the NLCD legend. Therefore, you can add the NLCD legend to the Map by using just one line of code (Map.add_legend).

    +

    Next, add the NLCD legend to the Map. The geemap package has several built-in legends, including the NLCD legend. Therefore, you can add the NLCD legend to the Map by using just one line of code (Map.add_legend).

    title = ‘NLCD Land Cover Classification’
    Map.add_legend(legend_title=title, builtin_legend=‘NLCD’)

    -

    Alternatively, if you want to add a custom legend, you can define a legend dictionary with labels as keys and colors as values, then you can use Map.add_legend to add the custom legend to the Map.  

    -

    legend_dict = {   ‘11 Open Water’: ‘466b9f’,   ‘12 Perennial Ice/Snow’: ‘d1def8’,   ‘21 Developed, Open Space’: ‘dec5c5’,   ‘22 Developed, Low Intensity’: ‘d99282’,   ‘23 Developed, Medium Intensity’: ‘eb0000’,   ‘24 Developed High Intensity’: ‘ab0000’,   ‘31 Barren Land (Rock/Sand/Clay)’: ‘b3ac9f’,   ‘41 Deciduous Forest’: ‘68ab5f’,   ‘42 Evergreen Forest’: ‘1c5f2c’,   ‘43 Mixed Forest’: ‘b5c58f’,   ‘51 Dwarf Scrub’: ‘af963c’,   ‘52 Shrub/Scrub’: ‘ccb879’,   ‘71 Grassland/Herbaceous’: ‘dfdfc2’,   ‘72 Sedge/Herbaceous’: ‘d1d182’,   ‘73 Lichens’: ‘a3cc51’,   ‘74 Moss’: ‘82ba9e’,   ‘81 Pasture/Hay’: ‘dcd939’,   ‘82 Cultivated Crops’: ‘ab6c28’,   ‘90 Woody Wetlands’: ‘b8d9eb’,   ‘95 Emergent Herbaceous Wetlands’: ‘6c9fb8’}
    +

    Alternatively, if you want to add a custom legend, you can define a legend dictionary with labels as keys and colors as values, then you can use Map.add_legend to add the custom legend to the Map.

    +

    legend_dict = { ‘11 Open Water’: ‘466b9f’, ‘12 Perennial Ice/Snow’: ‘d1def8’, ‘21 Developed, Open Space’: ‘dec5c5’, ‘22 Developed, Low Intensity’: ‘d99282’, ‘23 Developed, Medium Intensity’: ‘eb0000’, ‘24 Developed High Intensity’: ‘ab0000’, ‘31 Barren Land (Rock/Sand/Clay)’: ‘b3ac9f’, ‘41 Deciduous Forest’: ‘68ab5f’, ‘42 Evergreen Forest’: ‘1c5f2c’, ‘43 Mixed Forest’: ‘b5c58f’, ‘51 Dwarf Scrub’: ‘af963c’, ‘52 Shrub/Scrub’: ‘ccb879’, ‘71 Grassland/Herbaceous’: ‘dfdfc2’, ‘72 Sedge/Herbaceous’: ‘d1d182’, ‘73 Lichens’: ‘a3cc51’, ‘74 Moss’: ‘82ba9e’, ‘81 Pasture/Hay’: ‘dcd939’, ‘82 Cultivated Crops’: ‘ab6c28’, ‘90 Woody Wetlands’: ‘b8d9eb’, ‘95 Emergent Herbaceous Wetlands’: ‘6c9fb8’}
    title = ‘NLCD Land Cover Classification’
    Map.add_legend(legend_title=title, legend_dict=legend_dict)

    -

    The Map with the NLCD 2019 image and legend should look like Fig. F6.3.7.

    -

    -

    Fig. F6.3.7 The NLCD 2019 image layer displayed in geemap

    -

    The Map above shows only the NLCD 2019 image. To create an Earth Engine App for visualizing land cover change, we need a stack of NLCD images. Let’s print the list of system IDs of all available NLCD images.

    +

    The Map with the NLCD 2019 image and legend should look like Fig. F6.3.7.

    +
    +
    +

    +

    Fig. F6.3.7 The NLCD 2019 image layer displayed in geemap

    +
    +
    +

    The Map above shows only the NLCD 2019 image. To create an Earth Engine App for visualizing land cover change, we need a stack of NLCD images. Let’s print the list of system IDs of all available NLCD images.

    dataset.aggregate_array(‘system:id’).getInfo()

    The output should look like this.

    [‘USGS/NLCD_RELEASES/2019_REL/NLCD/2001’,
    @@ -1977,16 +2277,16 @@ Map.add_legend(legend_title=title, legend_dict=legend_dict)

    ‘USGS/NLCD_RELEASES/2019_REL/NLCD/2019’]

    Select the eight NLCD epochs after 2000.

    years = [‘2001’, ‘2004’, ‘2006’, ‘2008’, ‘2011’, ‘2013’, ‘2016’, ‘2019’]

    -

    Define a function for filtering the NLCD ImageCollection by year and select the ‘landcover’ band.

    +

    Define a function for filtering the NLCD ImageCollection by year and select the ‘landcover’ band.

    16 Get an NLCD image by year.

    -

    def getNLCD(year):   # Import the NLCD collection.   dataset = ee.ImageCollection(‘USGS/NLCD_RELEASES/2019_REL/NLCD’)   # Filter the collection by year.   nlcd = dataset.filter(ee.Filter.eq(‘system:index’, year)).first()    # Select the land cover band.   landcover = nlcd.select(‘landcover’);   return landcover

    -

    Create an NLCD ImageCollection to be used in the split-panel map.

    +

    def getNLCD(year): # Import the NLCD collection. dataset = ee.ImageCollection(‘USGS/NLCD_RELEASES/2019_REL/NLCD’) # Filter the collection by year. nlcd = dataset.filter(ee.Filter.eq(‘system:index’, year)).first() # Select the land cover band. landcover = nlcd.select(‘landcover’); return landcover

    +

    Create an NLCD ImageCollection to be used in the split-panel map.

    17 Create an NLCD image collection for the selected years.

    -

    collection = ee.ImageCollection(ee.List(years).map(lambda year: getNLCD(year)))

    +

    collection = ee.ImageCollection(ee.List(years).map(lambda year: getNLCD(year)))

    Print out the list of system IDs of the selected NLCD images covering the contiguous United States.

    collection.aggregate_array(‘system:id’).getInfo()

    The output should look like this.

    @@ -1995,11 +2295,11 @@ Map.add_legend(legend_title=title, legend_dict=legend_dict)

    ‘USGS/NLCD_RELEASES/2019_REL/NLCD/2006’,
    ‘USGS/NLCD_RELEASES/2019_REL/NLCD/2008’,
    ‘USGS/NLCD_RELEASES/2019_REL/NLCD/2011’,

    -

     ‘USGS/NLCD_RELEASES/2019_REL/NLCD/2013’,
    +

    ‘USGS/NLCD_RELEASES/2019_REL/NLCD/2013’,
    ‘USGS/NLCD_RELEASES/2019_REL/NLCD/2016’,
    ‘USGS/NLCD_RELEASES/2019_REL/NLCD/2019’]

    Next, create a list of labels to populate the dropdown list.

    -

    labels = [f’NLCD {year}’ for year in years]
    +

    labels = [f’NLCD {year}’ for year in years]
    labels

    The output should look like this.

    [‘NLCD 2001’,
    @@ -2008,15 +2308,19 @@ labels

    ‘NLCD 2008’,
    ‘NLCD 2011’,
    ‘NLCD 2013’,

    -

     ‘NLCD 2016’,
    +

    ‘NLCD 2016’,
    ‘NLCD 2019’]

    -

    The last step is to create a split-panel map by passing the NLCD ImageCollection and list of labels to Map.ts_inspector.

    +

    The last step is to create a split-panel map by passing the NLCD ImageCollection and list of labels to Map.ts_inspector.

    Map.ts_inspector(left_ts=collection, right_ts=collection, left_names=labels, right_names=labels)
    Map

    The split-panel map should look like Fig. F6.3.8.

    -

    -

    Fig. F6.3.8 A split-panel map for visualizing land cover change with geemap

    -

    To visualize land cover change, choose one NLCD image from the left dropdown list and another image from the right dropdown list, then use the slider to swipe through to visualize land cover change interactively. Click the close button in the bottom-right corner to close the split-panel map and return to the NLCD 2019 image shown in Fig. F6.3.7.

    +
    +
    +

    +

    Fig. F6.3.8 A split-panel map for visualizing land cover change with geemap

    +
    +
    +

    To visualize land cover change, choose one NLCD image from the left dropdown list and another image from the right dropdown list, then use the slider to swipe through to visualize land cover change interactively. Click the close button in the bottom-right corner to close the split-panel map and return to the NLCD 2019 image shown in Fig. F6.3.7.

    @@ -2027,61 +2331,97 @@ Note
    -

    Code Checkpoint F63c. The book’s repository contains information about what your code should look like at this point.

    +

    Code Checkpoint F63c. The book’s repository contains information about what your code should look like at this point.

    17.1 Publishing an Earth Engine App Using a Local Web Server

    -

    In this section, you will learn how to deploy an Earth Engine App using a local computer as a web server. Assume that you have completed Sect. 3 and created a Jupyter notebook named nlcd_app.ipynb. First, you need to download ngrok, a program that can turn your computer into a secure web server and connect it to the ngrok cloud service, which accepts traffic on a public address. Download ngrok from https://ngrok.com and unzip it to a directory on your computer, then copy  nlcd_app.ipynb to the same directory. Open the Anaconda Prompt (on Windows) or the Terminal (on macOS/Linux) and enter the following commands. Make sure you change /path/to/ngrok/dir to your computer directory where the ngrok executable is located, e.g., ~/Downloads.  

    +

    In this section, you will learn how to deploy an Earth Engine App using a local computer as a web server. Assume that you have completed Sect. 3 and created a Jupyter notebook named nlcd_app.ipynb. First, you need to download ngrok, a program that can turn your computer into a secure web server and connect it to the ngrok cloud service, which accepts traffic on a public address. Download ngrok from https://ngrok.com and unzip it to a directory on your computer, then copy nlcd_app.ipynb to the same directory. Open the Anaconda Prompt (on Windows) or the Terminal (on macOS/Linux) and enter the following commands. Make sure you change /path/to/ngrok/dir to your computer directory where the ngrok executable is located, e.g., ~/Downloads.

    cd /path/to/ngrok/dir

    conda activate gee
    -voila –no-browser nlcd_app.ipynb

    +voila –no-browser nlcd_app.ipynb

    The output of the terminal should look like this.

    -

    -

    Fig. F6.3.9 The output of the terminal running Voilà

    -

    Voilà can be used to run, convert, and serve a Jupyter notebook as a standalone app. Click the link (e.g., http://localhost:8866) shown in the terminal window to launch the interactive dashboard. Note that the port number is 8866, which is needed in the next step to launch ngrok. Open another terminal and enter the following command.

    +
    +
    +

    +

    Fig. F6.3.9 The output of the terminal running Voilà

    +
    +
    +

    Voilà can be used to run, convert, and serve a Jupyter notebook as a standalone app. Click the link (e.g., http://localhost:8866) shown in the terminal window to launch the interactive dashboard. Note that the port number is 8866, which is needed in the next step to launch ngrok. Open another terminal and enter the following command.

    cd /path/to/ngrok/dir

    ngrok http 8866

    -

    The output of the terminal should look like Fig. F6.3.10. Click the link shown in the terminal window to launch the interactive dashboard. The link should look like https://random-string.ngrok.io, which is publicly accessible. Anyone with the link will be able to launch the interactive dashboard and use the split-panel map to visualize NLCD land cover change. Keep in mind that the dashboard might take several seconds to load, so please be patient.

    -

    -

    Fig. F6.3.10 The output of the terminal running ngrok

    -

    To stop the web server, press Ctrl + C on both terminal windows. See below for some optional settings for running Voilà and ngrok.

    +

    The output of the terminal should look like Fig. F6.3.10. Click the link shown in the terminal window to launch the interactive dashboard. The link should look like https://random-string.ngrok.io, which is publicly accessible. Anyone with the link will be able to launch the interactive dashboard and use the split-panel map to visualize NLCD land cover change. Keep in mind that the dashboard might take several seconds to load, so please be patient.

    +
    +
    +

    +

    Fig. F6.3.10 The output of the terminal running ngrok

    +
    +
    +

    To stop the web server, press Ctrl + C on both terminal windows. See below for some optional settings for running Voilà and ngrok.

    To show code cells from your App, run the following from the terminal.

    voila –no-browser –strip_sources=False nlcd_app.ipynb

    To protect your App with a password, run the following.

    -

    ngrok http -auth=“username:password” 8866

    +

    ngrok http -auth=“username:password” 8866

    17.2 Publish an Earth Engine App Using Cloud Platforms

    -

    In this section, you will learn how to deploy an Earth Engine App on cloud platforms, such as Heroku and Google App Engine. Heroku is a “platform as a service” that enables developers to build, run, and operate applications entirely in the cloud. It has a free tier with limited computing hours, which would be sufficient for this lab. Follow the steps below to deploy the Earth Engine App on Heroku.

    -

    First, go to https://github.com/signup to sign up for a GitHub account if you don’t have one already. Once your GitHub account has been created, log into your account and navigate to the sample app repository: https://github.com/giswqs/earthengine-apps. Click the Fork button in the top-right corner to fork this repository into your account. Two important files in the repository are worth mentioning here: requirements.txt lists the required packages (e.g., geemap) to run the App, while Procfile specifies the commands that are executed by the App on startup. The content of Procfile should look like this.

    -

    web: voila –port=$PORT –no-browser –strip_sources=True –enable_nbextensions=True –MappingKernelManager.cull_interval=60 –MappingKernelManager.cull_idle_timeout=120 notebooks/

    -

    The above command instructs the server to hide the source code and periodically check for idle kernels—in this example, every 60 seconds—and cull them if they have been idle for more than 120 seconds. This can avoid idle kernels using up the server computing resources. The page served by Voilà will contain a list of any notebooks in the notebooks directory.

    -

    Next, go to https://signup.heroku.com to sign up for a Heroku account if you don’t have one already. Log into your account and click the New button in the top-right corner, then choose Create new app from the dropdown list (Fig. F6.3.11).

    -

    -

    Fig. F6.3.11 Creating a new app in Heroku

    +

    In this section, you will learn how to deploy an Earth Engine App on cloud platforms, such as Heroku and Google App Engine. Heroku is a “platform as a service” that enables developers to build, run, and operate applications entirely in the cloud. It has a free tier with limited computing hours, which would be sufficient for this lab. Follow the steps below to deploy the Earth Engine App on Heroku.

    +

    First, go to https://github.com/signup to sign up for a GitHub account if you don’t have one already. Once your GitHub account has been created, log into your account and navigate to the sample app repository: https://github.com/giswqs/earthengine-apps. Click the Fork button in the top-right corner to fork this repository into your account. Two important files in the repository are worth mentioning here: requirements.txt lists the required packages (e.g., geemap) to run the App, while Procfile specifies the commands that are executed by the App on startup. The content of Procfile should look like this.

    +

    web: voila –port=$PORT –no-browser –strip_sources=True –enable_nbextensions=True –MappingKernelManager.cull_interval=60 –MappingKernelManager.cull_idle_timeout=120 notebooks/

    +

    The above command instructs the server to hide the source code and periodically check for idle kernels—in this example, every 60 seconds—and cull them if they have been idle for more than 120 seconds. This can avoid idle kernels using up the server computing resources. The page served by Voilà will contain a list of any notebooks in the notebooks directory.

    +

    Next, go to https://signup.heroku.com to sign up for a Heroku account if you don’t have one already. Log into your account and click the New button in the top-right corner, then choose Create new app from the dropdown list (Fig. F6.3.11).

    +
    +
    +

    +

    Fig. F6.3.11 Creating a new app in Heroku

    +
    +

    Choose a name for your App (Fig. F6.3.12). Note that the App name must be unique; if an App name has already been taken, you won’t be able to use it.

    -

    -

    Fig. F6.3.12 Choosing an App name

    -

    Once the Heroku App has been created, click the Deploy tab and choose GitHub as the deployment method. Connect to your GitHub account and enter earthengine-apps in the search box. The repository should be listed beneath the search box. Click the Connect button to connect the repository to Heroku (Fig. F6.3.13).

    -

    -

    Fig. F6.3.13 Connecting a GitHub account to Heroku

    -

    Under the same Deploy tab, scroll down and click Enable Automatic Deploys (Fig. F6.3.14). This will enable Heroku to deploy a new version of the App whenever the GitHub repository gets updated.

    -

    -

    Fig. F6.3.14 Enabling Automatic Deploys on Heroku

    -

    Since using Earth Engine requires authentication, we need to set the Earth Engine token as an environment variable so that the web App can pass the Earth Engine authentication. If you have completed Sect. 3, you have successfully authenticated Earth Engine on your computer and the token can be found in the following file path, depending on the operating system you are using. Note that you might need to show the hidden directories on your computer in order to see the .config folder under the home directory.

    +
    +
    +

    +

    Fig. F6.3.12 Choosing an App name

    +
    +
    +

    Once the Heroku App has been created, click the Deploy tab and choose GitHub as the deployment method. Connect to your GitHub account and enter earthengine-apps in the search box. The repository should be listed beneath the search box. Click the Connect button to connect the repository to Heroku (Fig. F6.3.13).

    +
    +
    +

    +

    Fig. F6.3.13 Connecting a GitHub account to Heroku

    +
    +
    +

    Under the same Deploy tab, scroll down and click Enable Automatic Deploys (Fig. F6.3.14). This will enable Heroku to deploy a new version of the App whenever the GitHub repository gets updated.

    +
    +
    +

    +

    Fig. F6.3.14 Enabling Automatic Deploys on Heroku

    +
    +
    +

    Since using Earth Engine requires authentication, we need to set the Earth Engine token as an environment variable so that the web App can pass the Earth Engine authentication. If you have completed Sect. 3, you have successfully authenticated Earth Engine on your computer and the token can be found in the following file path, depending on the operating system you are using. Note that you might need to show the hidden directories on your computer in order to see the .config folder under the home directory.

    Windows: C:UsersUSERNAME.configearthenginecredentialsLinux: /home/USERNAME/.config/earthengine/credentials
    MacOS: /Users/USERNAME/.config/earthengine/credentials

    -

    Open the credentials file using a text editor. Select and copy the long string wrapped by the double quotes (Fig. F6.3.15). Do not include the double quotes in the copied string.

    -

    -

    Fig. F6.3.15 The Earth Engine authentication token

    -

    Next, navigate to your web App on Heroku. Click the Settings tab, then click Reveal Config Vars on the page. Enter EARTHENGINE_TOKEN as the key and paste the string copied above as the value. Click the Add button to set EARTHENGINE_TOKEN as an environment variable (Fig. F6.3.16) that will be used by geemap to authenticate Earth Engine.

    -

    -

    Fig. F6.3.16 Setting the Earth Engine token as an environment variable on Heroku

    -

    The last step is to commit some changes to your forked GitHub repository to trigger Heroku to build and deploy the App. If you are familiar with Git commands, you can push changes to the repository from your local computer to GitHub. If you have not used Git before, you can navigate to your repository on GitHub and make changes directly using the browser. For example, you can navigate to README.md and click the Edit icon on the page to start editing the file. Simply place the cursor at the end of the file and press Enter to add an empty line to the file, then click the Commit changes button at the bottom of the page to save changes. This should trigger Heroku to build the App. Check the build status under the latest activities of the Overview tab. Once the App has been built and deployed successfully, you can click the Open app button in the top-right corner to launch the web App (Fig. F6.3.17). When the App is open in your browser, click nlcd_app.ipynb to launch the split-panel map for visualizing land cover change. This is the same notebook that we developed in Sect. 3. The App URL should look like this: https://APP-NAME.herokuapp.com/voila/render/nlcd_app.ipynb.

    +

    Open the credentials file using a text editor. Select and copy the long string wrapped by the double quotes (Fig. F6.3.15). Do not include the double quotes in the copied string.

    +
    +
    +

    +

    Fig. F6.3.15 The Earth Engine authentication token

    +
    +
    +

    Next, navigate to your web App on Heroku. Click the Settings tab, then click Reveal Config Vars on the page. Enter EARTHENGINE_TOKEN as the key and paste the string copied above as the value. Click the Add button to set EARTHENGINE_TOKEN as an environment variable (Fig. F6.3.16) that will be used by geemap to authenticate Earth Engine.

    +
    +
    +

    +

    Fig. F6.3.16 Setting the Earth Engine token as an environment variable on Heroku

    +
    +
    +

    The last step is to commit some changes to your forked GitHub repository to trigger Heroku to build and deploy the App. If you are familiar with Git commands, you can push changes to the repository from your local computer to GitHub. If you have not used Git before, you can navigate to your repository on GitHub and make changes directly using the browser. For example, you can navigate to README.md and click the Edit icon on the page to start editing the file. Simply place the cursor at the end of the file and press Enter to add an empty line to the file, then click the Commit changes button at the bottom of the page to save changes. This should trigger Heroku to build the App. Check the build status under the latest activities of the Overview tab. Once the App has been built and deployed successfully, you can click the Open app button in the top-right corner to launch the web App (Fig. F6.3.17). When the App is open in your browser, click nlcd_app.ipynb to launch the split-panel map for visualizing land cover change. This is the same notebook that we developed in Sect. 3. The App URL should look like this: https://APP-NAME.herokuapp.com/voila/render/nlcd_app.ipynb.

    Congratulations! You have successfully deployed the Earth Engine App on Heroku.

    -

    -

    Fig. F6.3.17 Setting the Earth Engine token as an environment variable on Heroku

    +
    +
    +

    +

    Fig. F6.3.17 Setting the Earth Engine token as an environment variable on Heroku

    +
    +
    @@ -2092,32 +2432,32 @@ Note
    -

    Code Checkpoint F63d. The book’s repository contains information about what your code should look like at this point.

    +

    Code Checkpoint F63d. The book’s repository contains information about what your code should look like at this point.

    -

    Question 1. What are the pros and cons of designing Earth Engine Apps using geemap and ipywidgets, compared to the JavaScript Earth Engine User Interface API?

    -

    Question 2. What are the pros and cons of deploying Earth Engine Apps on Heroku, compared to a local web server and the Earth Engine Code Editor?

    +

    Question 1. What are the pros and cons of designing Earth Engine Apps using geemap and ipywidgets, compared to the JavaScript Earth Engine User Interface API?

    +

    Question 2. What are the pros and cons of deploying Earth Engine Apps on Heroku, compared to a local web server and the Earth Engine Code Editor?

    Synthesis

    -

    Assignment 1. Replace the NLCD datasets with other multitemporal land cover datasets (e.g., United States Department of Agriculture National Agricultural Statistics Service Cropland Data Layers, visible in the Earth Engine Data Catalog) and modify the web App for visualizing land cover change using the chosen land cover datasets. Deploy the web App using multiple platforms (i.e., JavaScript Code Editor, ngrok, and Heroku). More land cover datasets can be found at  https://developers.google.com/earth-engine/datasets/tags/landcover.

    +

    Assignment 1. Replace the NLCD datasets with other multitemporal land cover datasets (e.g., United States Department of Agriculture National Agricultural Statistics Service Cropland Data Layers, visible in the Earth Engine Data Catalog) and modify the web App for visualizing land cover change using the chosen land cover datasets. Deploy the web App using multiple platforms (i.e., JavaScript Code Editor, ngrok, and Heroku). More land cover datasets can be found at https://developers.google.com/earth-engine/datasets/tags/landcover.

    Conclusion

    -

    In this chapter, you learned how to design Earth Engine Apps using both the Earth Engine User Interface API (JavaScript) and geemap (Python). You also learned how to deploy Earth Engine Apps on multiple platforms, such as the JavaScript Code Editor, a local web server, and Heroku. The skill of designing and deploying interactive Earth Engine Apps is essential for making your research and data products more accessible to the scientific community and the general public. Anyone with the link to your web App can analyze and visualize Earth Engine datasets without needing an Earth Engine account.

    +

    In this chapter, you learned how to design Earth Engine Apps using both the Earth Engine User Interface API (JavaScript) and geemap (Python). You also learned how to deploy Earth Engine Apps on multiple platforms, such as the JavaScript Code Editor, a local web server, and Heroku. The skill of designing and deploying interactive Earth Engine Apps is essential for making your research and data products more accessible to the scientific community and the general public. Anyone with the link to your web App can analyze and visualize Earth Engine datasets without needing an Earth Engine account.

    References

    -

    Chapter F6.4: Combining R and Earth Engine

    +

    Chapter F6.4: Combining R and Earth Engine

    @@ -2130,18 +2470,17 @@ Chapter Information

    Author

    -

     

    Cesar Aybar, David Montero, Antony Barja, Fernando Herrera, Andrea Gonzales, and Wendy Espinoza

    Overview

    -

    The purpose of this chapter is to introduce rgee, a non-official API for Earth Engine. You will explore the main features available in rgee and how to set up an environment that integrates rgee with third-party R and Python packages. After this chapter, you will be able to combine R, Python, and JavaScript in the same workflow.

    +

    The purpose of this chapter is to introduce rgee, a non-official API for Earth Engine. You will explore the main features available in rgee and how to set up an environment that integrates rgee with third-party R and Python packages. After this chapter, you will be able to combine R, Python, and JavaScript in the same workflow.

    Learning Outcomes

    • Becoming familiar with rgee, the Earth Engine R API interface.
      • -
    • Integrating rgee with other R packages.
      • +
    • Integrating rgee with other R packages.
    • Displaying interactive maps.
    • Integrating Python and R packages using reticulate.
    • Combining Earth Engine JavaScript and Python APIs with R.
      • @@ -2151,7 +2490,7 @@ Chapter Information

      Assumes you know how to:

      • Install the Python environment (Chap. F6.3).
        • -
      • Use the require function to load code from existing modules (Chap. F6.1).
        • +
      • Use the require function to load code from existing modules (Chap. F6.1).
      • Use the basic functions and logic of Python.
      • Configure an environment variable and use .Renviron files.
      • Create Python virtual environments.
        • @@ -2162,8 +2501,8 @@ Chapter Information

    Introduction

    -

    R is a popular programming language established in statistical science with large support in reproducible research, geospatial analysis, data visualization, and much more. To get started with R, you will need to satisfy some extra software requirements. First, install an up-to-date R version (at least 4.0) in your work environment. The installation procedure will vary depending on your operating system (i.e., Windows, Mac, or Linux). Hands-On Programming with R (Garrett Grolemund 2014, Appendix A) explains step by step how to proceed. We strongly recommend that Windows users install Rtools to avoid compiling external static libraries.

    -

    If you are new to R, a good starting point is the book Geocomputation with R (Lovelace et al. 2019) or Spatial Data Science with Application in R (Pebesma and Bivand 2021). In addition, we recommend using an integrated development environment (e.g., Rstudio) or a code editor (e.g., Visual Studio Code) to create a suitable setting to display and interact with R objects.

    +

    R is a popular programming language established in statistical science with large support in reproducible research, geospatial analysis, data visualization, and much more. To get started with R, you will need to satisfy some extra software requirements. First, install an up-to-date R version (at least 4.0) in your work environment. The installation procedure will vary depending on your operating system (i.e., Windows, Mac, or Linux). Hands-On Programming with R (Garrett Grolemund 2014, Appendix A) explains step by step how to proceed. We strongly recommend that Windows users install Rtools to avoid compiling external static libraries.

    +

    If you are new to R, a good starting point is the book Geocomputation with R (Lovelace et al. 2019) or Spatial Data Science with Application in R (Pebesma and Bivand 2021). In addition, we recommend using an integrated development environment (e.g., Rstudio) or a code editor (e.g., Visual Studio Code) to create a suitable setting to display and interact with R objects.

    The following R packages must be installed (find more information in the R manual) in order to go through the practicum section.

    @@ -2181,13 +2520,17 @@ install.packages(‘raster’) # Reading, writing, manipulating, analyzing and m install.packages(‘magick’) # Advanced Graphics and Image-Processing in R
    install.packages(‘leaflet.extras2’) # Extra Functionality for leaflet
    install.packages(‘cptcity’) # colour gradients from the ‘cpt-city’ web archive

    -

    Earth Engine officially supports client libraries only for the JavaScript and Python programming languages. While the Earth Engine Code Editor offers a convenient environment for rapid prototyping in JavaScript, the lack of a mechanism for integration with local environments makes the development of complex scripts tricky. On the other hand, the Python client library offers much versatility, enabling support for third-party packages. However, not all Earth and environmental scientists code in Python. Hence, a significant number of professionals are not members of the Earth Engine community. In the R ecosystem, rgee (Aybar et al. 2020) tries to fill this gap by wrapping the Earth Engine Python API via reticulate (Kevin Ushey et al. 2021). The rgee package extends and supports all the Earth Engine classes, modules, and functions, working as fast as the other APIs.

    -

    -

    Fig. F6.4.1 A simplified diagram of rgee functionality

    -

    Figure F6.4.1 illustrates how rgee bridges the Earth Engine platform with the R ecosystem. When an Earth Engine request is created in R, rgee transforms this piece of code into Python. Next, the Earth Engine Python API converts the Python code into JSON. Finally, the JSON file request is received by the server through a Web REST API. Users could get the response using the getInfo method by following the same path in reverse.

    +

    Earth Engine officially supports client libraries only for the JavaScript and Python programming languages. While the Earth Engine Code Editor offers a convenient environment for rapid prototyping in JavaScript, the lack of a mechanism for integration with local environments makes the development of complex scripts tricky. On the other hand, the Python client library offers much versatility, enabling support for third-party packages. However, not all Earth and environmental scientists code in Python. Hence, a significant number of professionals are not members of the Earth Engine community. In the R ecosystem, rgee (Aybar et al. 2020) tries to fill this gap by wrapping the Earth Engine Python API via reticulate (Kevin Ushey et al. 2021). The rgee package extends and supports all the Earth Engine classes, modules, and functions, working as fast as the other APIs.

    +
    +
    +

    +

    Fig. F6.4.1 A simplified diagram of rgee functionality

    +
    +
    +

    Figure F6.4.1 illustrates how rgee bridges the Earth Engine platform with the R ecosystem. When an Earth Engine request is created in R, rgee transforms this piece of code into Python. Next, the Earth Engine Python API converts the Python code into JSON. Finally, the JSON file request is received by the server through a Web REST API. Users could get the response using the getInfo method by following the same path in reverse.

    -

    18.1 Installing rgee 

    -

    To run, rgee needs a Python environment with two packages: NumPy and earthengine-api. Because instructions change frequently, installation is explained at the following checkpoint:

    +

    18.1 Installing rgee

    +

    To run, rgee needs a Python environment with two packages: NumPy and earthengine-api. Because instructions change frequently, installation is explained at the following checkpoint:

    @@ -2198,7 +2541,7 @@ Note
    -

    Code Checkpoint F64a. The book’s repository contains information about setting up the rgee environment.

    +

    Code Checkpoint F64a. The book’s repository contains information about setting up the rgee environment.

    After installing both the R and Python requirements, you can now initialize your Earth Engine account from within R. Consider that R, in contrast to Javascript and Python, supports three distinct Google APIs: Earth Engine, Google Drive, and Google Cloud Storage (GCS).

    @@ -2211,7 +2554,7 @@ Note

    20 Initialize Earth Engine and GDee_Initialize(drive = TRUE)

    -

    If the Google account is verified and the permission is granted, you will be directed to an authentication token. Copy and paste this token into your R console. Consider that the GCS API requires setting up credentials manually; look up and explore the rgee vignette for more information. The verification step is only required once; after that, rgee saves the credentials in your system.

    +

    If the Google account is verified and the permission is granted, you will be directed to an authentication token. Copy and paste this token into your R console. Consider that the GCS API requires setting up credentials manually; look up and explore the rgee vignette for more information. The verification step is only required once; after that, rgee saves the credentials in your system.

    @@ -2222,62 +2565,66 @@ Note
    -

    Code Checkpoint F64b. The book’s repository contains information about what your code should look like at this point.

    +

    Code Checkpoint F64b. The book’s repository contains information about what your code should look like at this point.

    -

    20.1 Creating a 3D Population Density Map with rgee and rayshader

    -

    First, import the rgee, rayshader, and raster packages.

    +

    20.1 Creating a 3D Population Density Map with rgee and rayshader

    +

    First, import the rgee, rayshader, and raster packages.

    library(rayshader)
    library(raster)
    library(rgee)

    -

    Initialize the Earth Engine and Google Drive APIs using ee_Initialize. Both credentials must come from the same Google account.

    +

    Initialize the Earth Engine and Google Drive APIs using ee_Initialize. Both credentials must come from the same Google account.

    ee_Initialize(drive = TRUE)

    -

    Then, we will access the WorldPop Global Project Population Data dataset. In rgee, the Earth Engine spatial classes (ee\(Image, ee\)ImageCollection, and ee$FeatureCollection) have a special attribute called Dataset. Users can use it along with autocompletion to quickly find the desired dataset.

    +

    Then, we will access the WorldPop Global Project Population Data dataset. In rgee, the Earth Engine spatial classes (ee\(Image, ee\)ImageCollection, and ee$FeatureCollection) have a special attribute called Dataset. Users can use it along with autocompletion to quickly find the desired dataset.

    collections <- ee\(ImageCollection\)Dataset
    population_data <- collections\(CIESIN_GPWv411_GPW_Population_Density population_data_max <- population_data\)max()

    -

    If you need more information about the Dataset, use ee_utils_dataset_display to go to the official documentation in the Earth Engine Data Catalog.

    +

    If you need more information about the Dataset, use ee_utils_dataset_display to go to the official documentation in the Earth Engine Data Catalog.

    population_data %>% ee_utils_dataset_display()

    -

    The rgee package provides various built-in functions to retrieve data from Earth Engine (Aybar et al. 2020). In this example, we use ee_as_raster, which automatically converts an ee$Image (server object) into a RasterLayer (local object).

    +

    The rgee package provides various built-in functions to retrieve data from Earth Engine (Aybar et al. 2020). In this example, we use ee_as_raster, which automatically converts an ee$Image (server object) into a RasterLayer (local object).

    sa_extent <- ee\(Geometry\)Rectangle(
    - coords = c(-100, -50, -20, 12),
    - geodesic = TRUE,
    - proj = “EPSG:4326”)

    +coords = c(-100, -50, -20, 12),
    +geodesic = TRUE,
    +proj = “EPSG:4326”)

    population_data_ly_local <- ee_as_raster(
    - image = population_data_max,
    - region = sa_extent,
    - dsn = “/home/pc-user01/population.tif”, # change for your own path. scale = 5000
    +image = population_data_max,
    +region = sa_extent,
    +dsn = “/home/pc-user01/population.tif”, # change for your own path. scale = 5000
    )

    -

    Now, turn a RasterLayer into a matrix suitable for rayshader.

    +

    Now, turn a RasterLayer into a matrix suitable for rayshader.

    pop_matrix <- raster_to_matrix(population_data_ly_local)

    Next, modify the matrix population density values, adding:

      -
    • Texture, based on five colors (lightcolor, shadowcolor, leftcolor, rightcolor, and centercolor; see rayshader::create_texture documentation)
      • +
    • Texture, based on five colors (lightcolor, shadowcolor, leftcolor, rightcolor, and centercolor; see rayshader::create_texture documentation)
    • Color and shadows (rayshader::sphere_shade)

    pop_matrix %>%
    - sphere_shade(
    -   texture = create_texture(“#FFFFFF”, “#0800F0”, “#FFFFFF”, “#FFFFFF”, “#FFFFFF”)
    - ) %>%
    - plot_3d(
    -   pop_matrix,
    -   zoom = 0.55, theta = 0, zscale = 100, soliddepth = -24,
    -   solidcolor = “#525252”, shadowdepth = -40, shadowcolor = “black”,
    -   shadowwidth = 25, windowsize = c(800, 720)
    - )

    -

    Lastly, define a title and subtitle for the plot. Use rayshader::render_snapshot to export the final results (Fig. F6.4.2).

    -

    text <- paste0( “South Americanpopulation density”,
    - strrep(“n”, 27), “Source:GPWv411: Population Density (Gridded Population of the World Version 4.11)”)

    -

    render_snapshot(
    - filename = “30_poblacionsudamerica.png”,
    - title_text = text,
    - title_size = 20,
    - title_color = “black”,
    - title_font = “Roboto bold”,
    - clear = TRUE
    +sphere_shade(
    +texture = create_texture(“#FFFFFF”, “#0800F0”, “#FFFFFF”, “#FFFFFF”, “#FFFFFF”)
    +) %>%
    +plot_3d(
    +pop_matrix,
    +zoom = 0.55, theta = 0, zscale = 100, soliddepth = -24,
    +solidcolor = “#525252”, shadowdepth = -40, shadowcolor = “black”,
    +shadowwidth = 25, windowsize = c(800, 720)
    )

    -

    -

    Fig. F6.4.2 3D population density map of South America

    +

    Lastly, define a title and subtitle for the plot. Use rayshader::render_snapshot to export the final results (Fig. F6.4.2).

    +

    text <- paste0( “South Americanpopulation density”,
    +strrep(“n”, 27), “Source:GPWv411: Population Density (Gridded Population of the World Version 4.11)”)

    +

    render_snapshot(
    +filename = “30_poblacionsudamerica.png”,
    +title_text = text,
    +title_size = 20,
    +title_color = “black”,
    +title_font = “Roboto bold”,
    +clear = TRUE
    +)

    +
    +
    +

    +

    Fig. F6.4.2 3D population density map of South America

    +
    +
    @@ -2288,17 +2635,17 @@ Note
    -

    Code Checkpoint F64c. The book’s repository contains information about what your code should look like at this point.

    +

    Code Checkpoint F64c. The book’s repository contains information about what your code should look like at this point.

    20.2 Displaying Maps Interactively

    -

    Similar to the Code Editor, rgee supports the interactive visualization of spatial Earth Engine objects by Map$addLayer. First, let’s import the rgee and cptcity packages. The cptcity R package is a wrapper to the cpt-city color gradients web archive.

    +

    Similar to the Code Editor, rgee supports the interactive visualization of spatial Earth Engine objects by Map$addLayer. First, let’s import the rgee and cptcity packages. The cptcity R package is a wrapper to the cpt-city color gradients web archive.

    library(rgee)
    library(cptcity)
    ee_Initialize()

    -

    We’ll select an ee$Image; in this case, the Shuttle Radar Topography Mission 90 m (SRTM-90) Version 4.

    +

    We’ll select an ee$Image; in this case, the Shuttle Radar Topography Mission 90 m (SRTM-90) Version 4.

    dem <- ee\(Image\)Dataset$CGIAR_SRTM90_V4

    Then, we’ll set the visualization parameters as a list with the following elements.

      @@ -2307,72 +2654,92 @@ ee_Initialize()

    • palette: a list of CSS-style color strings

    viz <- list(
    - min = 600,
    - max = 6000,
    - palette = cpt(pal = ‘grass_elevation’, rev = TRUE)
    +min = 600,
    +max = 6000,
    +palette = cpt(pal = ‘grass_elevation’, rev = TRUE)
    )

    Then, we’ll create a simple display using Map$addLayer.

    m1 <- Map$addLayer(dem, visParams = viz, name = “SRTM”, shown = TRUE)

    -

    Optionally, you could add a custom legend using Map$addLayer (Fig. F6.4.3).

    +

    Optionally, you could add a custom legend using Map$addLayer (Fig. F6.4.3).

    pal <- Map$addLegend(viz)
    m1 + pal

    -

    -

    Fig. F6.4.3 Interactive visualization of SRTM-90 Version 4 elevation values

    -

    The procedure to display ee\(Geometry, ee\)Feature, and ee\(FeatureCollections objects is similar to the previous example effected on an ee\)Image. Users just need to change the arguments: eeObject and visParams.

    +
    +
    +

    +

    Fig. F6.4.3 Interactive visualization of SRTM-90 Version 4 elevation values

    +
    +
    +

    The procedure to display ee\(Geometry, ee\)Feature, and ee\(FeatureCollections objects is similar to the previous example effected on an ee\)Image. Users just need to change the arguments: eeObject and visParams.

    First, Earth Engine geometries (Fig. F6.4.4).

    vector <- ee\(Geometry\)Point(-77.011,-11.98) %>%
    - ee\(Feature\)buffer(50*1000)
    +ee\(Feature\)buffer(50*1000)
    Map\(centerObject(vector) Map\)addLayer(vector) # eeObject is a ee\(Geometry\)Polygon.

    -

    -

    Fig. F6.4.4 A polygon buffer surrounding the city of Lima, Peru

    +
    +
    +

    +

    Fig. F6.4.4 A polygon buffer surrounding the city of Lima, Peru

    +
    +

    Next, Earth Engine feature collections (Fig. F6.4.5).

    building <- ee\(FeatureCollection\)Dataset$
    - GOOGLE_Research_open-buildings_v1_polygon
    +GOOGLE_Research_open-buildings_v1_polygon
    Map\(setCenter(3.389, 6.492, 17) Map\)addLayer(building) # eeObject is a ee$FeatureCollection

    -

    -

    Fig. F6.4.5 Building footprints in Lagos, Nigeria

    -

    The rgee functionality also supports the display of ee\(ImageCollection via Map\)addLayers (note the extra “s” at the end). Map\(addLayers will use the same visualization parameters for all the images (Fig. F6.4.6). If you need different visualization parameters per image, use a Map\)addLayer within a for loop.

    +
    +
    +

    +

    Fig. F6.4.5 Building footprints in Lagos, Nigeria

    +
    +
    +

    The rgee functionality also supports the display of ee\(ImageCollection via Map\)addLayers (note the extra “s” at the end). Map\(addLayers will use the same visualization parameters for all the images (Fig. F6.4.6). If you need different visualization parameters per image, use a Map\)addLayer within a for loop.

    21 Define a ImageCollection

    -

    etp <- ee\(ImageCollection\)Dataset\(MODIS_NTSG_MOD16A2_105 %>%  ee\)ImageCollection\(select("ET") %>%  ee\)ImageCollection$filterDate(‘2014-04-01’, ‘2014-06-01’)

    +

    etp <- ee\(ImageCollection\)Dataset\(MODIS_NTSG_MOD16A2_105 %>% ee\)ImageCollection\(select("ET") %>% ee\)ImageCollection$filterDate(‘2014-04-01’, ‘2014-06-01’)

    22 Set viz paramsviz <- list(

    -

     min = 0,  max = 300,
    - palette = cpt(pal = “grass_bcyr”, rev = TRUE)
    +

    min = 0, max = 300,
    +palette = cpt(pal = “grass_bcyr”, rev = TRUE)
    )

    24 Estimate the median of the yearly composites from 2016 to 2020.

    median <- myclass\(get_year_composite()\)median()

    @@ -2438,13 +2811,17 @@ Map\(centerObject(roi) Map\)addLayer(roi)

    25 Estimate the median of the winter season composites from 2016 to 2020.

    wintermax <- myclass\(get_year_composite()\)select(‘winter’)$max()

    -

    We can display maps interactively using the Map$addLayer (Fig. F6.4.9), and use the leaflet layers control to switch between basemaps.

    +

    We can display maps interactively using the Map$addLayer (Fig. F6.4.9), and use the leaflet layers control to switch between basemaps.

    Map\(addLayer(wintermax, list(min = 0.1, max = 0.8), 'winterMax') | Map\)addLayer(median, list(min = 0.1, max = 0.8), ‘median’)

    -

    -

    Fig. F6.4.9 Comparison between the maximum historic winter NDVI and the mean historic NDVI. Colors represent the season when the maximum value occurred.

    +
    +
    +

    +

    Fig. F6.4.9 Comparison between the maximum historic winter NDVI and the mean historic NDVI. Colors represent the season when the maximum value occurred.

    +
    +

    And we can export the results to a GIF format.

    myclass$get_gif()

    -

    To get more information about the ndvi2gif package, visit its GitHub repository.

    +

    To get more information about the ndvi2gif package, visit its GitHub repository.

    @@ -2455,16 +2832,17 @@ Note
    -

    Code Checkpoint F64e. The book’s repository contains information about what your code should look like at this point.

    +

    Code Checkpoint F64e. The book’s repository contains information about what your code should look like at this point.

    -

    ## Converting JavaScript Modules to R

    -

    In recent years, the Earth Engine community has developed a lot of valuable third-party modules. Some incredible ones are geeSharp (Zuspan 2020), ee-palettes (Donchyts et al. 2020), spectral (Montero 2021), and LandsatLST (Ermida et al. 2020). While some of these modules have been implemented in Python and JavaScript (e.g., geeSharp and spectral), most are available only for JavaScript. This is a critical drawback, because it divides the Earth Engine community by programming languages. For example, if an R user wants to use tagee (Safanelli et al. 2020), the user will have to first translate the entire module to R.

    -

    In order to close this breach, the ee_extra Python package has been developed to unify the Earth Engine community. The philosophy behind ee_extra is that all of its extended functions, classes, and methods must be functional for the JavaScript, Julia, R, and Python client libraries. Currently, ee_extra is the base of the rgeeExtra (Aybar et al. 2021) and eemont (Montero 2021) packages.

    +
    +

    25.1 Converting JavaScript Modules to R

    +

    In recent years, the Earth Engine community has developed a lot of valuable third-party modules. Some incredible ones are geeSharp (Zuspan 2020), ee-palettes (Donchyts et al. 2020), spectral (Montero 2021), and LandsatLST (Ermida et al. 2020). While some of these modules have been implemented in Python and JavaScript (e.g., geeSharp and spectral), most are available only for JavaScript. This is a critical drawback, because it divides the Earth Engine community by programming languages. For example, if an R user wants to use tagee (Safanelli et al. 2020), the user will have to first translate the entire module to R.

    +

    In order to close this breach, the ee_extra Python package has been developed to unify the Earth Engine community. The philosophy behind ee_extra is that all of its extended functions, classes, and methods must be functional for the JavaScript, Julia, R, and Python client libraries. Currently, ee_extra is the base of the rgeeExtra (Aybar et al. 2021) and eemont (Montero 2021) packages.

    To demonstrate the potential of ee_extra, let’s study an example from the Landsat Land Surface Temperature (LST) JavaScript module. The Landsat LST module computes the land surface temperature for Landsat products (Ermida et al. 2020). First we will run it in the Earth Engine Code Editor; then we will replicate those results in R.

    -

    First, JavaScript. In a new script in the Code Editor, we must require the Landsat LST module.

    -

    var LandsatLST = require(   ‘users/sofiaermida/landsat_smw_lst:modules/Landsat_LST.js’);

    -

    The Landsat LST module contains a function named collection. This function receives the following parameters.

    +

    First, JavaScript. In a new script in the Code Editor, we must require the Landsat LST module.

    +

    var LandsatLST = require( ‘users/sofiaermida/landsat_smw_lst:modules/Landsat_LST.js’);

    +

    The Landsat LST module contains a function named collection. This function receives the following parameters.

    • The Landsat archive ID
    • The starting date of the Landsat collection
      • @@ -2473,34 +2851,38 @@ Note
    • A Boolean parameter specifying if we want to use the NDVI for computing a dynamic emissivity instead of using the emissivity from ASTER

    In the following code block, we are going to define all required parameters.

    -

    var geometry = ee.Geometry.Rectangle([-8.91, 40.0, -8.3, 40.4]);
    -var satellite = ‘L8’;
    -var date_start = ‘2018-05-15’;
    -var date_end = ‘2018-05-31’;
    -var use_ndvi = true;

    +

    var geometry = ee.Geometry.Rectangle([-8.91, 40.0, -8.3, 40.4]);
    +var satellite = ‘L8’;
    +var date_start = ‘2018-05-15’;
    +var date_end = ‘2018-05-31’;
    +var use_ndvi = true;

    Now, with all our parameters defined, we can compute the land surface temperature by using the collection method from Landsat LST.

    -

    var LandsatColl = LandsatLST.collection(satellite, date_start,
    -   date_end, geometry, use_ndvi);

    -

    The result is stored as an ImageCollection in the LandsatColl variable. Now select the first element of the collection as an example by using the first method.

    -

    var exImage = LandsatColl.first();

    +

    var LandsatColl = LandsatLST.collection(satellite, date_start,
    +date_end, geometry, use_ndvi);

    +

    The result is stored as an ImageCollection in the LandsatColl variable. Now select the first element of the collection as an example by using the first method.

    +

    var exImage = LandsatColl.first();

    This example image is now stored in a variable named ‘exImage’. Let’s display the LST result on the Map. For visualization purposes, we’ll define a color palette.

    -

    var cmap = [‘blue’, ‘cyan’, ‘green’, ‘yellow’, ‘red’];

    +

    var cmap = [‘blue’, ‘cyan’, ‘green’, ‘yellow’, ‘red’];

    Then, we’ll center the map in the region of interest.

    Map.centerObject(geometry);

    -

    Finally, let’s display the LST with the cmap color palette by using the Map.addLayer method (Fig. F6.4.10). This method receives the image to visualize, the visualization parameters, the color palette, and the name of the layer to show in the layer control. The visualization parameters will be:

    +

    Finally, let’s display the LST with the cmap color palette by using the Map.addLayer method (Fig. F6.4.10). This method receives the image to visualize, the visualization parameters, the color palette, and the name of the layer to show in the layer control. The visualization parameters will be:

    • min: 290 (a minimum LST value of 290 K)
    • max: 320 (a maximum LST value of 320 K)
      • -
    • palette: cmap (the color palette that was created some steps before)
      • +
    • palette: cmap (the color palette that was created some steps before)

    The name of the layer in the Map layer set will be LST.

    Map.addLayer(exImage.select(‘LST’), {
    -   min: 290,
    -   max: 320,
    -   palette: cmap
    +min: 290,
    +max: 320,
    +palette: cmap
    }, ‘LST’)

    -

    -

    Fig. F6.4.10 A map illustrating LST, obtained by following the JavaScript example

    +
    +
    +

    +

    Fig. F6.4.10 A map illustrating LST, obtained by following the JavaScript example

    +
    +
    @@ -2511,7 +2893,7 @@ Note
    -

    Code Checkpoint F64f. The book’s repository contains a script that shows what your code should look like at this point.

    +

    Code Checkpoint F64f. The book’s repository contains a script that shows what your code should look like at this point.

    Now, let’s use R to implement the same logic. As in the previous sections, import the R packages: rgee and rgeeExtra. Then, initialize your Earth Engine session.

    @@ -2521,7 +2903,7 @@ library(reticulate)

    ee_Initialize()

    Install rgeeExtra Python dependencies.

    py_install(packages = c(“regex”, “ee_extra”, “jsbeautifier”))

    -

    Using the function rgeeExtra::module loads the JavaScript module.

    +

    Using the function rgeeExtra::module loads the JavaScript module.

    LandsatLST <- module(“users/sofiaermida/landsat_smw_lst:modules/Landsat_LST.js”)

    The rest of the code is exactly the same as in JavaScript.

    geometry <- ee\(Geometry\)Rectangle(c(-8.91, 40.0, -8.3, 40.4))
    @@ -2530,8 +2912,12 @@ satellite <- ’L8’date_start <- ’2018-05-15’date_end <- ’2018- cmap <- c(‘blue’, ‘cyan’, ‘green’, ‘yellow’, ‘red’)

    lmod <- list(min = 290, max = 320, palette = cmap)
    Map\(centerObject(geometry) Map\)addLayer(exImage$select(‘LST’), lmod, ‘LST’)

    -

    -

    Fig. F6.4.11 A map illustrating LST, obtained by following the R example

    +
    +
    +

    +

    Fig. F6.4.11 A map illustrating LST, obtained by following the R example

    +
    +
    @@ -2542,14 +2928,15 @@ Note
    -

    Code Checkpoint F64g. The book’s repository contains information about what your code should look like at this point.

    +

    Code Checkpoint F64g. The book’s repository contains information about what your code should look like at this point.

    Question 1. When and why might users prefer to use R instead of Python to connect to Earth Engine?

    Question 2. What are the advantages and disadvantages of using rgee instead of the Earth Engine JavaScript Code Editor?

    +

    Synthesis

    -

    Assignment 1. Estimate the Gaussian curvature map from a digital elevation model using rgee and rgeeExtra. Hint: Use the module ‘users/joselucassafanelli/TAGEE:TAGEE-functions’.

    +

    Assignment 1. Estimate the Gaussian curvature map from a digital elevation model using rgee and rgeeExtra. Hint: Use the module ‘users/joselucassafanelli/TAGEE:TAGEE-functions’.

    Conclusion

    diff --git a/docs/F6/image22.png b/docs/F6/image22.png deleted file mode 100644 index d4fa308..0000000 Binary files a/docs/F6/image22.png and /dev/null differ diff --git a/docs/F6/image63.png b/docs/F6/image63.png deleted file mode 100644 index befe25a..0000000 Binary files a/docs/F6/image63.png and /dev/null differ diff --git a/docs/refineries.html b/docs/refineries.html index c9c6380..0e2389f 100644 --- a/docs/refineries.html +++ b/docs/refineries.html @@ -339,7 +339,7 @@ gtag('config', 'G-RK9ZLZQ6GL', { 'anonymize_ip': true});
    @@ -553,7 +553,7 @@ gtag('config', 'G-RK9ZLZQ6GL', { 'anonymize_ip': true});