ArcGIS Software: Importing and Manipulating Text Files and Images - Prof. T. Smith, Study notes of Electrical and Electronics Engineering

Download ArcGIS Software: Importing and Manipulating Text Files and Images - Prof. T. Smith and more Study notes Electrical and Electronics Engineering in PDF only on Docsity! NOTEBOOK FOR SPATIAL DATA ANALYSIS Part IV. Software for Analysis ______________________________________________________________________________________ ________________________________________________________________________ ESE 502 IV.1.2-1 Tony E. Smith 1.2. TIPS FOR USING ARCMAP 1.2.1 Importing Text Files to ArcMap (1) You can access information about importing text files using the HELP Index by writing delimited in the window and clicking: delimited text files → Accessing delimited text file data (2) Among the most important information is the following: ArcCatalog and ArcMap allow you to directly access data in delimited text files and work with them as tables. ArcCatalog, and the add data browser in ArcMap, lists files with a .txt, .asc, .csv, or .tab extension and assigns them a file type of Text File. a. Files with a .txt, .asc or .csv extension are interpreted as comma delimited, while files with a .tab extension are interpreted as tab delimited. b. If your text file is tab delimited, you should rename the extension as .tab. NOTE: If the .txt file extension is hidden (as it is by default in the Lab) then you must reveal this extension by the following procedure. • Open the directory containing the text file in “My Computer”, and in the main menu click: Tools → Folder Options → View • Now uncheck the box next to “Hide file extensions for known file types”. • You will then be able to see the .txt file extension and to rename this file with a .tab extension. c. If it is comma delimited you should make sure that all field names are in quotes( “name”). NOTE: You must NOT start a field name with a number. Otherwise the file will not be properly read in either ArcMap or ArcCatalog. (This important detail is missing from the HELP files !!) NOTEBOOK FOR SPATIAL DATA ANALYSIS Part IV. Software for Analysis ______________________________________________________________________________________ ________________________________________________________________________ ESE 502 IV.1.2-2 Tony E. Smith 1.2.2 Changing Path Directories in Map Documents When a red “!” appears where a map should be displayed, it usually means that your maps are in the wrong directory. To change all such maps with a single operation: (1) Left click on the red “!” next to the first file in the map document that has such a mark. (2) In the Data Source window that opens, select the appropriate data source file in the new directory. (3) Click “Add”, and the source directories of all files in the same directory as the first file will automatically be changed to the corresponding source file in the new directory (assuming that they exist). (4) If the map document contains source files in more than one directory, some files will still have a red “!”. So left click on any one of these, and repeat the above procedure. 1.2.3 Making a Column of Row Numbers in an Attribute Table Here there are two procedures that one can use: 1. Modify FID Numbers If the FID column of string identifiers is present, and in the correct order, then you can simply construct a numerical field that adds “1” to each FID number as follows: (1) Open the Attribute Table and select Options → Add Field. (2) Set Name = “ROW” and leave all else as is. Click OK. (3) In the new column that appears (with zero values) right click on the ROW label and select Calculate Values. (Ignore any warnings) (4) In the Field Calculator that opens, fill in the window labeled “ROW =” with the phrase “[FID] + 1” (no quotes), and click OK. The column should now be filled with the desired row numbers. 2. Construct an Independent Numbering An alternative procedure that is more flexible is to use the utility, rec_Number.cal, in the class directory, Sys502\Arcview\Projects\Utilities, as follows: NOTEBOOK FOR SPATIAL DATA ANALYSIS Part IV. Software for Analysis ______________________________________________________________________________________ ________________________________________________________________________ ESE 502 IV.1.2-5 Tony E. Smith (5) To display the Mask in a proper form, double click on its colored box in the Table of Contents, set Fill Color = “White”, and set the Outline Width and Outline Color to whatever you like. (6) Once again display the “Boundary” layer and enlarge it with the “Fixed Zoom In” tool to fill the window. Be sure that the “Mask” layer is positioned above it in the Table of Contents. ⇒ Any raster or contour output from subsequent analyses of point data inside the map will now automatically be clipped at the boundary as long as those displays are positioned below the “Mask” layer in the Table of Contents. 1.2.5 Making Spline Contours in Spatial Analyst (1) Select a point-data layer with values “X” to be contoured. (2) On the Spatial Analyst tool bar Click: Spatial Analyst → Interpolate to Raster → Spline (3) Then in the Spline Window set: • Z value field = X • Spline Type = Regularized • Weight: (0.1 = close fit to data → 5 = smoothed fit) → All other settings are OK. (4) If a raster grid of colors is OK, then you are done. • Right click on the layer (which is only a temporary file) and click Save as a Layer file . • To edit the displayed number of contours or their colors, right click on saved layer, click Properties → Symbology, and edit “Classes” or colors. (5) If you want contour curves rather than a raster grid, you can now use the “Surface Analysis” tool as follows. Select the raster-grid layer and click: Spatial Analyst → Surface Analysis → Contour NOTEBOOK FOR SPATIAL DATA ANALYSIS Part IV. Software for Analysis ______________________________________________________________________________________ ________________________________________________________________________ ESE 502 IV.1.2-6 Tony E. Smith (6) In the Contour Window choose the contour interval size you want (and check the Total number of contour values below to see if the number is adequate, but not too dense). • When you have made a contour map that is acceptable, right click on the layer (which is again a temporary file) and click Save as a Layer file . 1.2.6 Excluding Values from Map Displays To exclude a range of values from the map display of a given variable (layer): (1) Right click on the appropriate layer and click: Properties → Quantities → Classify → Exclusion (2) In the Data Exclusion Properties dialogue • Double click on the variable (so that it appears in the text box) • Specify the range to be excluded, such as: “Variable” <= -100 • Click OK (3) On the map, those features (points, choropleth areas) with value in this range will no longer be shown. 1.2.7 Importing ArcMap Images to the Web Arcmap images can be imported to the web by converting them to .gif files. One simple procedure for doing so is the following: (1) First export image from ARCMAP as an Enhanced Meta File (.emf) (2) Open the image in POWER POINT an edit in any way desired (you can even combine several images into one). (3) Save in POWER POINT to the Desktop as, say image, using File → Save as Web Page. (4) You will now see a file on the Desktop, image.htm, together with a folder, image_files. NOTEBOOK FOR SPATIAL DATA ANALYSIS Part IV. Software for Analysis ______________________________________________________________________________________ ________________________________________________________________________ ESE 502 IV.1.2-7 Tony E. Smith (5) If you open the folder you will see one .gif file with a label something like slide_0001. This is the file containing the desired image (open it to check). (6) Now copy this file to the same local directory where you keep the copy of your web page, and rename it as say image.gif. (7) You can then insert this into your web page by editing the page in Netscape using File → Edit Page, and then using Insert → Image. (8) In the “Image Properties” window that opens, just type in the name of the file, image.gif, and it will be inserted. (9) It is often easier to control the placement of the image by first creating a table in the web page using, Table → Insert Table, and then inserting image.gif into an appropriate cell of the table. (10) Finally, you can publish the web page in the usual way, by FTPing the image file to same directory on the server where your web page is kept. 1.2.8 Adding Areas to Map Polygons The following instructions are adapted from the help file in ARCMAP: 1. Open the attribute table of the layer containing the polygons. 2. If there is no field for area values, add a new field for area by clicking the Options button and selecting the new field option. Name the field “Area” and set Type = Double. 3. Click Calculate Values 4. Check Advanced. 5. Type the following VBA statement in the first text box: Dim dblArea as double Dim pArea as IArea Set pArea = [shape] dblArea = pArea.area 6. Type the variable dblArea in the text box directly under the area field name. 7. Click OK. NOTEBOOK FOR SPATIAL DATA ANALYSIS Part IV. Software for Analysis ______________________________________________________________________________________ ________________________________________________________________________ ESE 502 IV.1.2-10 Tony E. Smith 4. To make the transformation you will use the Visual Basic function, CDbl(.) , that converts Strings of numbers to Double-valued data. In the Field Calculator box labeled ‘TRACT_NUM = ’ type: CDbl([TRACT]) 5. Now click OK, and the new set of Double-valued data should appear in the column TRACT_NUM of the attribute table. Procedure for Long Numbers The above procedure will work for integers as long as you have no more than seven digits. If you have a long integer, like an STFID identification number for a Philadelphia city block, say “420171001021001” then you will not be able to reproduce this reliably as a number. The format “Double” only allow 7 places before the decimal (and “Long Integer” only allows 9 places). So it will be recorded in scientific notation, like “4.20171e+014”. These are not reliable for matching identifiers in join operations. So if the number displayed is in scientific notation, then the following additional steps can be used to correct this: 6. In the Table of Contents, right-click on the layer with the Attribute Table of interest and select: Properties → Fields. 7. Suppose you are converting STFID to a numerical field called STFID_NUM. Then in the “Layer Properties” window that opens, scroll down to the STFID_NUM row, and in the “Number Format” column click on the gray button with 3 dots. 8. In the “Number Format” window that opens, select Category = “Custom”, and select String Format = “0”. Click OK, and in the Layer Properties window click Apply and then OK. 9. If you check the Attribute Table again, then the full digit number for STFID_NUM should now be displayed. NOTE 1. The same procedure can be used to convert integer strings to numerical integers by designate the data type of the column as ‘Short Integer’ and using the Visual Basic function CInt(.) in place of CDbl(.). NOTE 2. There is also a VBA function CStr(.) which will convert numbers to strings. This may sometimes also be useful NOTEBOOK FOR SPATIAL DATA ANALYSIS Part IV. Software for Analysis ______________________________________________________________________________________ ________________________________________________________________________ ESE 502 IV.1.2-11 Tony E. Smith 1.2.12 Displaying Proper Distance Units When boundary shapefiles are in unknown units, there is usually problem getting the distance measurement tool to show distance values in the same units as the unknown coordinates. To accomplish this: 1. Open the Properties of the given Data Frame, and select Coordinate System. 2. Under Select a coordinate system, choose Layers and select either the boundary layer for the data frame (i.e., with polygon features) or any layer with point features. Click Apply and OK. 3. Now reopen Properties and select General. In the Units window you should now see that both Map and Display are active. 4. First set Display to “Unknown Units” (top of the list) and then set Map to “Unknown Units”. Click Apply and OK. 5. In the map display, the Distance Tool should now be displaying the same units as the unknown coordinates. 1.2.13 Editing Point Styles in ARCMAP For maximum flexibility in choosing colored points with or without borders, the following procedure is useful. For sake of concreteness, suppose you wish to enlarge a portion of a point pattern for an export display, and want to increase the size of dots and put a heavy black border around each for emphasis (so that they will print well in WORD). 1. Enlarge the desired portion in the window, and click Properties → Symbology. 2. Now in the Layer Properties window right click on any symbol and select Properties for All Symbols. 3. Select Circle 2 (the filled circle) and choose the size you want. 4. Now click on Properties. In the window that opens you will see a display of Layers to the left. If you click on the border layer (hollow circle) you can set its properties. (i) In the Properties section set Type = “Simple Marker Symbol”. (ii) Set Color = “No Color”, check “Outline” and Outline Color = “Black”. (iii) You will then see the border outline in the Preview window, and can reset both the Size and Outline Size to obtain a border with the desired thickness NOTEBOOK FOR SPATIAL DATA ANALYSIS Part IV. Software for Analysis ______________________________________________________________________________________ ________________________________________________________________________ ESE 502 IV.1.2-12 Tony E. Smith around the point. 5. Now click OK in each window, to get back to the Layer Properties window. Here you will see that points and borders have the right size, but not ramp colors. (i) To apply ramp colors (if desired) right click on any symbol and choose Ramp Colors. The ramp color window should now display the same solid color as the points. (ii) Click on the ramp color arrow and select the desired color ramp. These colors should now appear on the point symbols. 6. Click Apply and OK. You should then see these point styles in the map window. 7. Export this map, and close the map document without saving it (unless you want to keep these enlarged points as permanent choices). 1.2.14 Exporting Maps from ARCMAP to WORD To export a map that is currently displayed in ARCMAP, use the command File → Export Map 1. The single best format to use for exporting is Enhanced Meta File (.emf). This will give you good reproduction quality and will allow you maximum flexibility in editing the image in WORD. 2. These images can be imported to WORD using Insert → Picture → From File. 3. Be aware that some printers will not have map marker fonts. If your map marker symbols are not coming out, or are completely different in appearance, the best approach is to re-export the map from ARCMAP using the Format Options. In the Export Map window click “Options” at the bottom, and open the “Format” tag. Now check the option: Convert Marker Symbols to Polygons. This will convert your symbols to graphic objects that can be printed without font sets. 1.2.15 Making Legends for Exported Maps 1. The procedure in ARCMAP for composing maps with legends, north arrows, etc. is their “Layout View”. However a more flexible procedure (in my view) is simply to export each component as a “map” and arrange them using the Picture and Drawing toolbars in WORD. NOTEBOOK FOR SPATIAL DATA ANALYSIS Part IV. Software for Analysis ______________________________________________________________________________________ ________________________________________________________________________ ESE 502 IV.1.2-15 Tony E. Smith 1.2.17 Running Local G* Tests of Concentration in ARCMAP There is an option in ARCMAP for testing spatial concentration using local G*-statistics (denoted here by Gi*). The following notes illustrate the application of this test to the Eire Blood Group Data, and compare it with the corresponding MATLAB results. The testing tool in ARCMAP offers a number of weight-matrix options, but most importantly, allows one to import any weight matrix in the right format. So a second objective of these notes is to show how to construct matrices in the right format using MATLAB. 1. The first task is to open F:\sys502\arcview\projects\eire\Eire.mxd in ARCMAP, and check that the Attribute Table of the shapefile, Eire.shp, contains a field that identifies counties with the corresponding row numbers of these counties in MATLAB data. This Attribute Table already has a field called “ID” that contains these numbers. (For other applications you will need to construct such an identifier field, say “ID”. To do so, use the “Add Field” option in the Attribute Table. Set Name = “ID” and Type = “Short Integer”. Finally use the “Calculate Values” option and set ID = [FID] +1.) 2. The next task is to convert the exponential weight matrix used in Assignment 6 to ARCMAP format in order to construct a comparable test to that of Assignment 6. The following instructions for forming this matrix are essentially the same as in Assignment 6: Open MATLAB and import the file F:\sys502\matlab\eire.txt. You should now have a 26× 7 matrix, eire, in the workspace. 1. Define the centroid locations, L, and blood-group percentages, x , by: » L = eire(:,1:2); » x = eire(:,3); 2. Now make an exponential weight matrix by writing: » OUT = dist_wts(L,4,10,1); » W = OUT{1}; Here the argument 4 denotes the “exponential-weight-matrix” option, 10 denotes the exponent value, and 1 denotes the “include-diagonal-terms” option. 3. We now convert this weight matrix, W, to ARCMAP format using the MATLAB program arcmap_wt_matrix.m. Here the appropriate command is: >> W_list = arcmap_wt_matrix(W); The matrix, W_list, is actually a “list” form of W. To see this, display the first four rows of W_list by writing: NOTEBOOK FOR SPATIAL DATA ANALYSIS Part IV. Software for Analysis ______________________________________________________________________________________ ________________________________________________________________________ ESE 502 IV.1.2-16 Tony E. Smith >> W_list(1:4,:) ans = 1 1 0.30159 1 2 0.0036059 1 3 0.0051854 1 4 0.0056317 For example, the third line says that the (row-normalized) exponential weight of County 1 with County 3 is 0.0051854. This can also be verified by displaying the (1,3) element of W as: >> W(1,3) ans = 0.0051854 4. We next save this file and convert it to a format that can be read by ARCMAP as follows. First save W_list as a text file using >> save W_list.txt W_list –ascii Next we need to “clean” this file in EXCEL. Open W_list.txt in EXCEL and use Format → Cells → Number to format the first two columns as integers (Decimal places = 0 ) and the third column as, say, a 4-digit decimal (Decimal places = 4 ). Save as file as a tab delimited text file, W_list.txt in your home directory, which we shall assume to be S:\Home\eire. 5. Finally, recall from (1) above that ARCMAP needs to match the ID numbers in the first two columns of W_list.txt with a field numbering the Counties in Eire.shp. Since there is such a field, “ID”, we shall use this field. Open W_list.txt as a text file and add “ID” as a top line to this file, so that the first few lines now read: ID 1 1 0.3016 1 2 0.0036 1 3 0.0052 6. With this text file completed, we are now ready to carry out Gi* tests in ARCMAP. To do again open the map document file F:\sys502\arcview\projects\eire\eire.mxd in ARCMAP, and also open ArcToolBox. Then navigate to: NOTEBOOK FOR SPATIAL DATA ANALYSIS Part IV. Software for Analysis ______________________________________________________________________________________ ________________________________________________________________________ ESE 502 IV.1.2-17 Tony E. Smith Spatial Statistics Tools → Mapping Clusters → Hot Spot Analysis (Getis-Ord Gi*) 7. In the window that opens, set Input Feature Class = “Eire.shp”, Input Field = “BLOOD_GRP” Output Feature Class = “Eire_G_Arcmap.shp” 8. To use the custom weight matrix above, set: Conceptualization of Spatial Relationships = “Get Spatial Weights from File” Distance Band = “0” Weights Matrix File = “S:\Home\eire\W_list.txt” where it is assumed that the last line contains the proper path to W_list.txt. Leave all other options as is and click OK. 9. A new layer, “Eire_G_Arcmap”, will now appear in the Table of Contents. If you open the Attribute Table you will see that the last field is labeled “GiW_list”. The values in this column are standardized Gi* values, i.e., z-values of the form: Zi = (Gi* - E(Gi*))/std(Gi*) [where exact expressions for the mean, E(Gi*) and standard deviation, std(Gi*) can be found in the Getis-Ord paper listed in the reserve section of the class web page.] To see the relevance of these values, recall that under the null hypothesis of the random permutation test, the underlying x-values (in this case “Blood Group A” percentages for each county) are assumed to be typical random permutations of these values among counties. Ord has shown that under this hypothesis, the Zi-values above are asymptotically normally distributed for large sample sizes, n. [Here there are only n = 26 counties, so this asymptotic assumption is questionable at best. But nonetheless, it will be seen below that this test provides very reasonable results in the present example.] 10. To display the results of this test in a manner that is comparable to the random permutation test, g_perm_loc.m, in MATLAB, recall that this is a one-sided test focusing only on significant concentration levels. The appropriate z-values corresponding to one-sided significance levels in this case are given by: P-value 0.10 0.05 0.01 z-value 1.28 1.65 2.33 Hence to display these z-values (“GiW_list”) use a classification with Classes = “4” , Method = “Manual”, and values “1.28”, “1.65”, “2.33”, and “max” (where in this NOTEBOOK FOR SPATIAL DATA ANALYSIS Part IV. Software for Analysis ______________________________________________________________________________________ ________________________________________________________________________ ESE 502 IV.1.2-20 Tony E. Smith (j) Save this file as, say POINTS.shp. You can then join this to the polygon file by right clicking on the POLYGON file and selecting: Joins and Relates → Join (k) Here the best procedure for joining points to polygons is to change the “What do you want to joint to this layer?” option to read Join data from another layer based on spatial location 1.2.19 Saving Map Documents with Relative Paths When saving map documents it is useful to save only relative paths, so that projects can be moved to new directories without changing all paths to shapefiles, etc. This option can be set in ARCMAP by clicking: File → Document Properties Then at the bottom of the “Properties” window click: Data Source Options… → Store Relative Path Names Finally, if you check “Make relative paths the default” then this option will be used for all future map documents as well. 1.2.20 Increasing Unique Values for editing Raster Outputs (in Version 9.3) When you use the command Spatial Analyst → Interpolate to Raster → Spline to interpolate a set of point measurements (like temperatures at weather station locations in South America), you will obtain a default “class” representation of the interpolation (as with older versions). But when you right click on properties and try to edit this representation (change the number of classes or colors, etc.) it says: "Unique histogram does not exist. Do you want to compute values?" When you try to do this, it will tell you that there are too many unique values. This can be fixed by increasing the bound on unique values as follows: (1) In the main menu click: Tools → Options → Raster (2) In the raster window you will see an additional set of tabs about half way down. Click “Raster Attribute Table” (3) Here change the upper bound on unique values to, say, 500000. This should be enough. NOTEBOOK FOR SPATIAL DATA ANALYSIS Part IV. Software for Analysis ______________________________________________________________________________________ ________________________________________________________________________ ESE 502 IV.1.2-21 Tony E. Smith Now when you are asked if you want to compute unique values say “Yes”. The computation takes a few seconds, and then the desired editing window opens up.