Features and capabilities

Deep learning raster analysis tools enable you to use more than the standard machine learning classification techniques.

• Use convolutional neural networks for image classification.
• Use a deep learning model to either classify image pixels or detect objects such as airplanes, trees, vehicles, water bodies, and oil well pads.
• Integrate external deep learning model frameworks, including Keras, TensorFlow, and CNTK.
• Use a model definition file multiple times to detect change over time or detect objects in different areas of interest.
• Generate a polygon feature class showing detected objects to be used for additional analysis or workflows.
• The tools can use the CPU for distributed processing, or use the powerful GPU on each server node if available.

Get started with deep learning

The creation and export of training samples are done within ArcGIS Pro by a knowledgeable image analyst because they require strong knowledge of image classification workflows. The deep learning model is trained outside of ArcGIS Pro using a deep learning framework by a data scientist, or someone experienced in training deep learning models. Once the model is trained and defined in an Esri model definition file .emd, you share a dlpk item to Portal for ArcGIS and run the raster analysis tools to detect objects or classify pixels from Map Viewer, ArcGIS API for Python, ArcGIS REST API, or ArcGIS Pro. In some cases, all three steps may be completed by one analyst that has experience with deep learning models and ArcGIS image classification.

You will need to install the proper deep learning framework Python API (TensorFlow, CNTK, or Keras) in the ArcGIS Pro Python environment; otherwise, an error will occur when you add the Esri Model Definition file to the tool. Obtain the appropriate framework information from the creator of the Esri Model Definition file.

You will need to install the proper deep learning framework ArcGIS API for Python (TensorFlow, CNTK, or Keras) in the ArcGIS Server runtime Python environment; otherwise, an error will occur when you add the dlpk item into the tool. Obtain the appropriate framework information from the creator of the Esri Model Definition file

Refer to the framework's documentation for help with installation.

1. Create and export training samples.
   1. Use the Training Samples Manager to select or create a classification schema.
   2. Create training site samples for the class categories or features of interest. Save the training sample file.
   3. Run the Export Training Data for Deep Learning raster analysis tool to convert the source imagery and training samples into deep learning training data. The source imagery may be an image service. The output of the tool is the data store string where the output image chips, labels and metadata files will be stored. An additional output of the tool is a template .emd file to be populated by the data scientist.
2. Train the deep learning model.
   1. Use the image chips generated in step 1 above to train a model using a deep learning framework such as TensorFlow, CNTK, PyTorch or Keras.
   2. Update the .emd file with the name of the deep learning framework, the model configuration, and other optional properties. For information on required and optional parameters in the .emd file, see Esri model definition file.
3. Run the inference raster analysis tools from Map Viewer, ArcGIS API for Python, ArcGIS REST API or ArcGIS Pro.
   1. Use the Detect Objects Using Deep Learning or the Classify Pixels Using Deep Learning raster analysis tools to process your imagery. If the trained model incorporated custom Python raster functions with additional variables, such as padding or confidence threshold for fine-tuning the sensitivity, these variables will appear on the ArcGIS Pro or Map Viewer raster analysis tool's dialog box for user input. The data type, such as string or numeric, is specified in the Python raster function. Ideally, additional inference parameters should be limited to two.

      The output of the Detect Objects Using Deep Learning tool is a feature class showing the objects detected by the model, and the Classify Pixels Using Deep Learning tool outputs a classified raster.

      Note:

      The deep learning model framework must be installed on the machine where the ArcGIS Server is installed in order to run the inference tools because the inference logic is built into the framework Python API and is required for computation.

Esri model definition file

The .emd file is a JSON file that describes the trained deep learning model. It contains model definition parameters that are required to run the inference tools, and it should be modified by the data scientist that trained the model. There are required and optional parameters in the file as described in the table below.

Below is an example of a model definition file (.emd) that uses a standard model configuration:

{
    "Framework": "TensorFlow",
    "ModelConfiguration": "ObjectDetectionAPI",
    "ModelFile":"\\tree_detection.pb",
    "ModelType":"ObjectionDetection",
    "ImageHeight":850,
    "ImageWidth":850,
    "ExtractBands":[0,1,2],
    "Classes" : [
      {
        "Value": 0,
        "Name": "Tree",
        "Color": [0, 255, 0]
      }
    ]
}

Deep learning package (dlpk) item

Deep learning raster analysis tools require a deep learning model package (dlpk) as input. A deep learning model package is composed of the Esri model definition JSON file (.emd), the deep learning binary model file, and optionally, the Python raster function to be used.

When you have all components ready, you can compress all files into a .zip file, and upload the .zip file to your portal as a dlpk item. For more details, please refer to Add files from your computer. You can also change your .zip file extension to .dlpk so that when adding item the dlpk item type will be automatically detected.