Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference --...

97
Beyond Where: Modeling Spatial Relationships and Making Predictions Lauren Bennett Jenora D’Acosta Alberto Nieto Flora Vale esriurl.com/spatialstats

Transcript of Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference --...

Page 1: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions

Beyond Where: Modeling Spatial Relationships and Making PredictionsLauren Bennett

Jenora D’Acosta

Alberto Nieto

Flora Vale

esriurl.com/spatialstats

Page 2: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions
Page 3: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions
Page 4: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions
Page 5: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions
Page 6: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions
Page 7: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions
Page 8: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions
Page 9: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions
Page 10: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions
Page 11: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions
Page 12: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions
Page 13: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions
Page 14: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions
Page 15: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions
Page 16: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions
Page 17: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions
Page 18: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions
Page 19: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions

ε

Page 20: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions
Page 21: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions
Page 22: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions
Page 23: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions
Page 24: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions
Page 25: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions
Page 26: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions
Page 27: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions
Page 28: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions
Page 29: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions
Page 30: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions
Page 31: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions
Page 32: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions
Page 33: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions
Page 34: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions
Page 35: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions
Page 36: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions
Page 37: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions
Page 38: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions
Page 39: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions
Page 40: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions
Page 41: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions
Page 42: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions
Page 43: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions
Page 44: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions

Page 45: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions
Page 46: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions
Page 47: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions
Page 48: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions
Page 49: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions
Page 50: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions

“… the measurement of a relationship

the measurement

is taken.”

depends on where

Page 51: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions

Y

two variables

X

determine relationship significance and type

relationships across geography

Page 52: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions

… what does it mean for two variables to be

related to each other?

Page 53: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions

“good” relationships

Page 54: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions

“good” relationships

null relationships

Page 55: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions

low entropy

higher entropy

null relationships

“good” relationships

Page 56: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions

variable Y

low entropy

Page 57: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions

variable Y

high entropy

Page 58: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions

variable Y variable X

Page 59: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions

variable Y variable Xmutual

information

Page 60: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions

minimum spanning trees

low entropy

higher entropy

Page 61: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions

…how do we know if the relationship is

SIGNIFICANT???

Page 62: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions

Permutation-based distribution estimation

Page 63: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions

feature

Page 64: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions

each feature has two values

Y X

dependent explanatory

Page 65: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions

relationshipbetween and x

is evaluated for a neighborhood

Y X

Page 66: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions

relationshipbetween and x

is evaluated for a neighborhood

Y X

Page 67: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions

neighborhood distribution of variables minimum spanning tree entropy value

Page 68: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions

what is the probability that the observed entropyexists while and are

actually independent from each other?

Y X

Page 69: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions

observed local entropy

0.56

Page 70: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions

0.56

start a permutation

Page 71: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions

0.56

the dependent values are shuffled, while explanatory values are kept the same

Y

X

Page 72: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions

0.56

minimum spanning trees and entropy are calculated for the permutation

Page 73: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions

0.56

0.64

minimum spanning trees and entropy are calculated for the permutation

Page 74: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions

permutations

Y

X

Y

X

0.61

Y

X

Y

X

0.64

Y

X

Y

X

0.57

0.56

Y

X

Y

X

………

neighborhood

Page 75: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions

permutations

permutation entropy distribution

neighborhoodY

X

Y

X

0.61

Y

X

Y

X

0.64

Y

X

Y

X

0.57

0.56

Y

X

Y

X

………

Page 76: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions

observed entropy is converted to a p-value

permutation entropies

0.55 0.65

p-value = 0.0012

observed entropy0.56

Page 77: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions

entropy helps determine if relationships are significant,

but it does not tell uswhat type of relationship exists

Page 78: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions

classifying the local relationships

Page 79: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions

Positive Linear Negative Linear Convex Concave

Undefined Complex Not Significant

Page 80: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions

start with the distribution of the significant feature’s

neighborhood

Page 81: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions

estimate an ordinary linear regression model and

calculate AICc

Page 82: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions

square of X

estimate a second linear regression model

using the square of the explanatory variable

Page 83: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions

AICc values are compared, and a model is selected

X

Y Y

square of X

linear model quadratic model

AICc: -42.3 AICc: -51.4

Page 84: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions

is the R2 value greater

than 0.05?

Yes

No

which model was chosen?

linear

quadratic

sign of coefficient?

sign of coefficient?

positive

negative

positive

negative

Positive Linear

Negative Linear

Convex

ConcaveUndefined Complex

Page 85: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions

entropy

classify type

Page 86: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions

Positive Linear

Page 87: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions

Positive Linear

Page 88: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions
Page 89: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions

Convex

Page 90: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions

Convex

Page 91: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions
Page 92: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions

Negative Linear

Page 93: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions

Negative Linear

Page 94: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions

Positive Linear

Negative Linear

Concave

Convex

Undefined Complex

Not Significant

Page 95: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions

Demo

Page 96: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions
Page 97: Beyond Where: Modeling Spatial Relationships and Making ... · 2019 Esri User Conference -- Presentation, 2019 Esri UC, Beyond Where: Modeling Spatial Relationships and Making Predictions