Estimating and Simulating [-3pt] a SIRD Model of...

Post on 15-Aug-2020

2 views 0 download

Transcript of Estimating and Simulating [-3pt] a SIRD Model of...

Estimating and Simulating

a SIRD Model of COVID-19

Jesus Fernandez-Villaverde and Chad Jones

April 22, 2020

(Preliminary and incomplete)

0 / 58

Outline

• Basic model

◦ Social distancing via a time-varying β

• Estimation and simulation

◦ Different countries, U.S. states, and New York City

◦ Robustness to parameters

◦ “Forecasts” from each of the last 7 days

• Re-opening and herd immunity

◦ How much can we relax social distancing?

1 / 58

Basic Model

2 / 58

Notation

• Number of people who are (stocks):

S = Susceptible

I = Infectious

R = Recovered

D = Dead

• Constant population size is N

St + It + Rt + Dt = N

3 / 58

SIRD Model: Overview

• Susceptible get infected at rate βIt/N

New infections = βIt/N · St

• Infections resolve at Poisson rate γ, so the average number of

days until resolution is 1/γ so γ = .2 ⇒ 5 days.

• Resolution happens in one of two ways:

◦ Death: fraction δ

◦ Recovery: fraction 1 − δ

4 / 58

SIRD Model: Laws of Motion

∆St+1 = −βStIt/N︸ ︷︷ ︸

new infections

∆It+1 = βStIt/N︸ ︷︷ ︸

new infections

− γIt︸︷︷︸

resolving infections

∆Rt+1 = (1 − δ)γIt︸ ︷︷ ︸

recover

∆Dt+1 = δγIt︸︷︷︸

die

R0 = D0 = 0

5 / 58

Recyled notation (terrible) R0: Initial infection rate

• Initial reproduction number R0 ≡ β/γ

R0 = β × 1/γ

# of infections

from one sick

person

# of lengthy

contacts per

day

# of days

contacts are

infectious

• R0 = expected number of infections via the first sick person

◦ R0 > 1 ⇒ disease initially grows

◦ R0 < 1 ⇒ disease dies out: infectious generate less than 1

new infection

• If 1/γ = 5, then easy to have R0 >> 1

6 / 58

Basic Properties of Differential System (Hethcote 2000)

• Let st ≡ St/N = fraction susceptible

• If R0st > 1, the disease spreads, otherwise declines

• Initial exponential growth rate is β − γ

• As t → ∞, the total fraction of people ever infected, e∗, solves

(assuming s0 ≈ 1)

e∗ = −1

R0log(1 − e∗)

Long run is pinned down by R0 (and death rate),

γ affects timing

7 / 58

Social Distancing

• What about a time-varying infection rate βt?

◦ Disease characteristics – fixed, homogeneous

◦ Regional factors (NYC vs Montana) – fixed, heterogeneous

◦ Social distancing – varies over time and space

• Reasons why βt may change over time

◦ Policy changes on social distancing

◦ Individuals voluntarily change behavior to protect

themselves and others

◦ Superspreaders get infected quickly but then recover and

“burn out” early

8 / 58

Model of Social Distancing

• Assume two key parameters β0 and β∗

• Economy decays exponentially from β0 to β∗ at rate λ:

βt = β0e−λt + β∗(1 − e−λt)

⇒ can think about initial R0 = β0/γ

R0(t) = βt/γ and final R∗

0 = β∗/γ

• Interpretation:

◦ λ governs the rate of convergence to R∗

0

9 / 58

Estimates and Simulations

10 / 58

Estimation: Countries and States

• Parameters that are fixed and homogeneous

◦ γ = 0.2: average duration is 5 days (or γ = 0.1)

◦ δ = 0.003

Apr 1: 15% of mothers giving birth in NYC infected

With δ = .004, model says only 13.6% ever infected on Apr 1

• Parameters that vary across countries/states

◦ β0 and β∗

◦ λ: speed at which you move to β∗

◦ I0: initial number of infections (gets timing right)

• Objective function:

◦ Equally weighted SSR for Cumulative deaths (logs) and

Daily deaths (logs)11 / 58

Estimation based entirely on death data

• Excess death issue

◦ New York City added 3000+ deaths on April 15 ≈ 45% more

◦ The Economist ⇒ increases based on vital records

⇒ We adjust all NYC deaths before April 15 by this 45%

and non-NYC deaths upwards by 33%

• We use 5-day moving averages (centered)

◦ Otherwise, very serious “weekend effects” in which deaths

are underreported

◦ Even zero sometimes, followed by a large spike

12 / 58

Guide to Graphs

• 7 days of forecasts: Rainbow color order!

ROY-G-BIV (old to new, low to high)

◦ Black=current

◦ Red = oldest, Orange = second oldest, Yellow =third oldest...

◦ Violet (purple) = one day earlier

• For robustness graphs, same idea

◦ Black = baseline (e.g. δ = .003)

◦ Red = lowest parameter value (e.g. δ = .002)

◦ Green = highest parameter value (e.g. δ = .004)

13 / 58

Guide to Graphs (continued)

• R0 in subtitle: R0 / R0(today) / R∗

0

◦ Initial / Today / Final

• “%Infect”

◦ Today / t+30 / Final

◦ This is the percent ever infected

◦ (so fraction δ will eventually be deaths)

14 / 58

New York City: Cumulative Deaths per Million (δ = .003/.002/.004)

Mar 16 Mar 23 Mar 30 Apr 06 Apr 13 Apr 20 Apr 27

2020

0

200

400

600

800

1000

1200

1400

1600

1800

Cu

mu

lati

ve

dea

ths

per

mil

lio

n p

eop

leNew York City (only)

R0=3.4/1.4/0.9 = 0.003, =0.05, =0.2, %Infect=64/70/70

DATA THROUGH 21-APR-2020

δ = .003/.004 fit equally well in levels

δ = .002 cannot fit: 1700 deaths per

million people already means nearly

100% infected!

15 / 58

New York City: Daily Deaths per Million People (δ = .003/.002/.004)

Mar 17 Mar 31 Apr 14 Apr 28 May 12 May 26 Jun 09

2020

0

10

20

30

40

50

60

70

80

90

100

Dai

ly d

eath

s p

er m

illi

on

peo

ple

New York City (only)

R0=3.4/1.4/0.9 = 0.003, =0.05, =0.2, %Infect=64/70/70

DATA THROUGH 21-APR-2020

δ = .003/.004 fit equally well

16 / 58

New York City: Cumulative Deaths per Million (δ = .003/.002/.004)

Mar 16 Mar 30 Apr 13 Apr 27 May 11 May 25 Jun 08

2020

0

500

1000

1500

2000

2500

Cu

mu

lati

ve

dea

ths

per

mil

lio

n p

eop

leNew York City (only)

R0=3.4/1.4/0.9 = 0.003, =0.05, =0.2, %Infect=64/70/70

= 0.003

= 0.002

= 0.004

DATA THROUGH 21-APR-2020

But very different futures!

17 / 58

Spain: Cumulative Deaths per Million People (γ = .2/.1)

Mar 05 Mar 12 Mar 19 Mar 26 Apr 02 Apr 09 Apr 16 Apr 23

2020

0

100

200

300

400

500

600

700

Cum

ula

tive

dea

ths

per

mil

lion p

eople

Spain

R0=4.6/1.4/0.6 = 0.003, =0.07, =0.2, %Infect=21/23/23

DATA THROUGH 21-APR-2020

γ = .2 fits slightly better in levels

18 / 58

Spain: Daily Deaths per Million People (γ = .2/.1)

Mar 06 Mar 20 Apr 03 Apr 17 May 01 May 15 May 29

2020

0

5

10

15

20

25

30

Dai

ly d

eath

s per

mil

lion p

eople

Spain

R0=4.6/1.4/0.6 = 0.003, =0.07, =0.2, %Infect=21/23/23

DATA THROUGH 21-APR-2020

γ = .2 better fit for daily deaths

19 / 58

Spain: Cumulative Deaths per Million (Future, γ = .2/.1)

Mar 05 Mar 19 Apr 02 Apr 16 Apr 30 May 14 May 28

2020

0

100

200

300

400

500

600

700

800

Cu

mu

lati

ve

dea

ths

per

mil

lio

n p

eop

leSpain

R0=4.6/1.4/0.6 = 0.003, =0.07, =0.2, %Infect=21/23/23

= 0.2

= 0.1

DATA THROUGH 21-APR-2020

Very different futures!

20 / 58

Italy: Daily Deaths per Million People (γ = .2/.1)

Feb 24 Mar 09 Mar 23 Apr 06 Apr 20 May 04 May 18

2020

0

2

4

6

8

10

12

14

16

18

20

Dai

ly d

eath

s per

mil

lion p

eople

Italy

R0=4.4/1.8/0.8 = 0.003, =0.07, =0.2, %Infect=19/22/22

DATA THROUGH 21-APR-2020

γ = .1 gives a flatter right tail of

deaths

21 / 58

Repeated “Forecasts” from the

past 7 days of data

– After peak, forecasts settle down.

– Before that, very noisy!

22 / 58

Spain (7 days): Daily Deaths per Million People

Mar 06 Mar 20 Apr 03 Apr 17 May 01 May 15 May 29

2020

0

5

10

15

20

25

30

Dai

ly d

eath

s per

mil

lion p

eople

Spain

R0=4.6/0.7/0.6 = 0.003, =0.07, =0.2, %Infect=21/23/23

DATA THROUGH 21-APR-2020

23 / 58

Italy (7 days): Daily Deaths per Million People

Feb 24 Mar 09 Mar 23 Apr 06 Apr 20 May 04 May 18

2020

0

2

4

6

8

10

12

14

16

18

20

Dai

ly d

eath

s per

mil

lion p

eople

Italy

R0=4.4/0.9/0.8 = 0.003, =0.07, =0.2, %Infect=19/22/22

DATA THROUGH 21-APR-2020

24 / 58

New York City (7 days): Daily Deaths per Million People

Mar 17 Mar 31 Apr 14 Apr 28 May 12 May 26 Jun 09

2020

0

10

20

30

40

50

60

70

80

90

Dai

ly d

eath

s p

er m

illi

on

peo

ple

New York City (plus)

R0=3.5/1.2/0.5 = 0.003, =0.04, =0.2, %Infect=53/59/60

DATA THROUGH 21-APR-2020

25 / 58

New York City (7 days): Cumulative Deaths per Million (Future)

Mar 16 Mar 30 Apr 13 Apr 27 May 11 May 25 Jun 08

2020

0

200

400

600

800

1000

1200

1400

1600

1800

2000

Cu

mu

lati

ve

dea

ths

per

mil

lio

n p

eop

leNew York City (plus)

R0=3.5/1.2/0.5 = 0.003, =0.04, =0.2, %Infect=53/59/60

DATA THROUGH 21-APR-2020

26 / 58

California (7 days): Cumulative Deaths per Million

Mar 06 Mar 13 Mar 20 Mar 27 Apr 03 Apr 10 Apr 17 Apr 24

2020

0

5

10

15

20

25

30

35

40

45

50

Cum

ula

tive

dea

ths

per

mil

lion p

eople

California

R0=4.0/1.2/1.1 = 0.003, =0.08, =0.2, %Infect= 2/ 6/ 8

DATA THROUGH 21-APR-2020

27 / 58

California (7 days): Daily Deaths per Million People

Mar 07 Mar 21 Apr 04 Apr 18 May 02 May 16 May 30

2020

0

1

2

3

4

5

6

7

8

9

10

Dai

ly d

eath

s per

mil

lion p

eople

California

R0=4.0/1.2/1.1 = 0.003, =0.08, =0.2, %Infect= 2/ 6/ 8

DATA THROUGH 21-APR-2020

(noisy and unreliable)

28 / 58

California (7 days): Cumulative Deaths per Million (Future)

Mar 06 Mar 20 Apr 03 Apr 17 May 01 May 15 May 29

2020

0

50

100

150

200

250

300

Cu

mu

lati

ve

dea

ths

per

mil

lio

n p

eop

leCalifornia

R0=4.0/1.2/1.1 = 0.003, =0.08, =0.2, %Infect= 2/ 6/ 8

DATA THROUGH 21-APR-2020

(noisy and unreliable)

29 / 58

U.K. (7 days): Cumulative Deaths per Million

Mar 07 Mar 14 Mar 21 Mar 28 Apr 04 Apr 11 Apr 18 Apr 25

2020

0

50

100

150

200

250

300

350

400

450

Cu

mu

lati

ve

dea

ths

per

mil

lio

n p

eop

leUnited Kingdom

R0=3.8/1.0/0.5 = 0.003, =0.04, =0.2, %Infect=13/20/20

DATA THROUGH 21-APR-2020

30 / 58

U.K. (7 days): Daily Deaths per Million People

Mar 08 Mar 22 Apr 05 Apr 19 May 03 May 17 May 31

2020

0

5

10

15

20

25

Dai

ly d

eath

s per

mil

lion p

eople

United Kingdom

R0=3.8/1.0/0.5 = 0.003, =0.04, =0.2, %Infect=13/20/20

DATA THROUGH 21-APR-2020

(noisy and unreliable)

31 / 58

U.K. (7 days): Cumulative Deaths per Million (Future)

Mar 07 Mar 21 Apr 04 Apr 18 May 02 May 16 May 30

2020

0

100

200

300

400

500

600

700

800

Cu

mu

lati

ve

dea

ths

per

mil

lio

n p

eop

leUnited Kingdom

R0=3.8/1.0/0.5 = 0.003, =0.04, =0.2, %Infect=13/20/20

DATA THROUGH 21-APR-2020

(noisy and unreliable)

32 / 58

Sweden (7 days): Daily Deaths per Million People

Mar 14 Mar 28 Apr 11 Apr 25 May 09 May 23 Jun 06

2020

0

2

4

6

8

10

12

14

16

Dai

ly d

eath

s p

er m

illi

on

peo

ple

Sweden

R0=3.1/1.0/0.5 = 0.003, =0.05, =0.2, %Infect= 9/13/13

DATA THROUGH 21-APR-2020

(noisy and unreliable)

33 / 58

France (7 days): Daily Deaths per Million People

Feb 18 Mar 03 Mar 17 Mar 31 Apr 14 Apr 28 May 12

2020

0

5

10

15

20

25

Dai

ly d

eath

s per

mil

lion p

eople

France

R0=7.4/0.9/0.5 = 0.003, =0.05, =0.2, %Infect=16/20/20

DATA THROUGH 21-APR-2020

(noisy and unreliable)

34 / 58

Cumulative Deaths per Million, Log Scale

0 10 20 30 40 50 60 0.1

0.5

1

2

4

8

16

32

64

128

256

512

1024

2048

Doubles every 10 days, g=7%Doubles e

very 5 days, g=14%

Dou

bles

eve

ry 3

day

s, g=

23%

Dou

bles

eve

ry 2

day

s, g

=35

%

Iran

Korea, South

SwedenUnited Kingdom

FranceItalySpain

United States

DATA THROUGH 21-APR-2020

DAYS SINCE 1 DEATH PER 10 MILLION PEOPLE

CUMULATIVE DEATHS PER MILLION

Italy seems a better guide to France/U.K.

35 / 58

S. Korea (7 days): Daily Deaths per Million People

Feb 23 Mar 08 Mar 22 Apr 05 Apr 19 May 03 May 17

2020

0

0.02

0.04

0.06

0.08

0.1

0.12

0.14

0.16

0.18

Dai

ly d

eath

s p

er m

illi

on

peo

pleKorea, South

R0=1.5/0.8/0.5 = 0.003, =0.02, =0.2, %Infect= 0/ 0/ 0

DATA THROUGH 21-APR-2020

36 / 58

Washington (7 days): Daily Deaths per Million People

Mar 03 Mar 17 Mar 31 Apr 14 Apr 28 May 12 May 26

2020

0

1

2

3

4

5

6

Dai

ly d

eath

s p

er m

illi

on

peo

pleWashington

R0=1.8/1.0/0.5 = 0.003, =0.02, =0.2, %Infect= 4/ 6/ 6

DATA THROUGH 21-APR-2020

(noisy and unreliable)

37 / 58

Louisiana (7 days): Daily Deaths per Million People

Mar 17 Mar 31 Apr 14 Apr 28 May 12 May 26 Jun 09

2020

0

5

10

15

20

25

Dai

ly d

eath

s p

er m

illi

on

peo

ple

Louisiana

R0=3.1/0.9/0.8 = 0.003, =0.08, =0.2, %Infect=15/20/21

DATA THROUGH 21-APR-2020

38 / 58

Florida (7 days): Daily Deaths per Million People

Mar 11 Mar 25 Apr 08 Apr 22 May 06 May 20 Jun 03

2020

0

0.5

1

1.5

2

2.5

3

3.5

Dai

ly d

eath

s per

mil

lion p

eople

Florida

R0=3.9/1.0/0.5 = 0.003, =0.05, =0.2, %Infect= 2/ 3/ 3

DATA THROUGH 21-APR-2020

(noisy and unreliable)

39 / 58

Detriot (Wayne County, 7 days): Daily Deaths per Million People

Mar 21 Apr 04 Apr 18 May 02 May 16 May 30 Jun 13

2020

0

10

20

30

40

50

60

Dai

ly d

eath

s p

er m

illi

on

peo

ple

Detroit

R0=2.9/1.0/0.5 = 0.003, =0.05, =0.2, %Infect=34/41/41

DATA THROUGH 21-APR-2020

40 / 58

Massachusetts (7 days): Daily Deaths per Million People

Mar 23 Apr 06 Apr 20 May 04 May 18 Jun 01 Jun 15

2020

0

10

20

30

40

50

60

Dai

ly d

eath

s p

er m

illi

on

peo

pleMassachusetts

R0=3.4/1.6/1.6 = 0.003, =0.20, =0.2, %Infect=15/56/63

DATA THROUGH 21-APR-2020

(noisy and unreliable)

41 / 58

District of Columbia (7 days): Daily Deaths per Million People

Mar 23 Apr 06 Apr 20 May 04 May 18 Jun 01 Jun 15

2020

0

10

20

30

40

50

60

70

Dai

ly d

eath

s p

er m

illi

on

peo

ple

District of Columbia

R0=1.8/1.3/0.5 = 0.003, =0.02, =0.2, %Infect= 9/27/30

DATA THROUGH 21-APR-2020

(noisy and unreliable)

42 / 58

Reopening and Herd Immunity

43 / 58

Percent Ever Infected would be very informative

— Percent Ever Infected (today) —

δ = .002 δ = .003 δ = .004

New York City (plus) 78 53 41

Detroit 51 34 26

Spain 32 21 16

Italy 28 19 14

Michigan 25 16 12

France 24 16 12

Massachusetts 23 15 12

United Kingdom 20 13 10

Sweden 13 9 6

District of Columbia 13 9 7

New York excluding NYC 8 5 4

Denmark 5 3 2

Germany 4 3 2

Florida 3 2 2

California 3 2 1

Tennessee 2 1 144 / 58

Herd Immunity

• How far can we relax social distancing?

• Let s(t) = S(t)/N = the fraction still susceptible

◦ The disease will die out as long as

R0(t)s(t) < 1

◦ That is, if the “new” R0 is smaller than 1/s(t)

◦ Today’s infected people infect fewer than 1 person on

average

• We can relax social distancing to raise R0(t) to 1/s(t)

45 / 58

Herd Immunity and Opening the Economy?

Percent R0(t+30) PercentSusceptible with no way back

R0 R0(t) t+30 outbreak to normal

New York City 3.5 1.2 40.5 2.5 55.6

Detroit 2.9 1.0 58.7 1.7 34.8

Spain 4.6 0.7 77.4 1.3 15.1

Italy 4.4 0.9 78.4 1.3 10.9

Michigan 3.4 1.0 76.3 1.3 13.3

France 7.4 0.9 79.8 1.3 5.1

Massachusetts 3.4 1.6 43.8 2.3 38.2

United Kingdom 3.8 1.0 80.5 1.2 8.4

Sweden 3.1 1.0 87.0 1.1 8.0

District of Columbia 1.8 1.3 73.4 1.4 4.2

New York excluding NYC 3.5 0.9 92.9 1.1 8.3

Washington 1.8 1.0 93.9 1.1 13.1

Denmark 2.7 0.7 96.4 1.0 16.1

Germany 4.6 0.9 95.3 1.0 3.7

Florida 3.9 1.0 96.6 1.0 2.9

California 4.0 1.2 94.1 1.1 -4.3

Korea, South 1.5 0.8 99.8 1.0 33.646 / 58

Simulations of Re-Opening

• Begin with the basic estimates shown already

• Different policies are then adopted starting around May 20

◦ Black: assumes R0(today) remains in place forever

◦ Red: assumes R0(suppress)= 1/s(today)

◦ Green: we move 25% of the way from R0(today) back to

initial R0 = “normal”

◦ Purple: we move 50% of the way from R0(today) back to

initial R0 = “normal”

• We assume these R0 values stay in place forever

◦ In practice, over course, β would likely start to fall as

mortality rises

47 / 58

Spain: Re-Opening

Apr May Jun Jul Aug Sep

2020

0

10

20

30

40

50

60

70

80

Dai

ly d

eath

s per

mil

lion p

eople

Spain

R0(t)=0.7, R

0(suppress)=1.3, R

0(25/50)=1.7/2.6, = 0.003, =0.07

48 / 58

Italy: Re-Opening

Mar Apr May Jun Jul Aug Sep

2020

0

10

20

30

40

50

60

70

80

90

Dai

ly d

eath

s per

mil

lion p

eople

Italy

R0(t)=0.9, R

0(suppress)=1.3, R

0(25/50)=1.8/2.6, = 0.003, =0.07

49 / 58

New York City: Re-Opening

Apr May Jun Jul Aug Sep

2020

0

10

20

30

40

50

60

70

80

90

Dai

ly d

eath

s per

mil

lion p

eople

New York City (plus)

R0(t)=1.2, R

0(suppress)=2.5, R

0(25/50)=1.8/2.3, = 0.003, =0.04

NYC has herd immunity and can

re-open significantly in a month!

(???)

50 / 58

New York excluding NYC: Re-Opening

Apr May Jun Jul Aug Sep Oct

2020

0

10

20

30

40

50

60

70

80

90

Dai

ly d

eath

s per

mil

lion p

eople

New York excluding NYC

R0(t)=0.9, R

0(suppress)=1.1, R

0(25/50)=1.5/2.2, = 0.003, =0.07

But not the rest of NY state!

51 / 58

California: Re-Opening

Apr May Jun Jul Aug Sep

2020

0

20

40

60

80

100

120

140

Dai

ly d

eath

s p

er m

illi

on

peo

ple

California

R0(t)=1.2, R

0(suppress)=1.1, R

0(25/50)=1.8/2.4, = 0.003, =0.07

52 / 58

Sweden: Re-Opening

Apr May Jun Jul Aug Sep

2020

0

10

20

30

40

50

60

Dai

ly d

eath

s per

mil

lion p

eople

Sweden

R0(t)=1.0, R

0(suppress)=1.2, R

0(25/50)=1.5/2.0, = 0.003, =0.05

53 / 58

Detroit: Re-Opening

Apr May Jun Jul Aug Sep Oct

2020

0

10

20

30

40

50

60

Dai

ly d

eath

s per

mil

lion p

eople

Detroit

R0(t)=1.0, R

0(suppress)=1.7, R

0(25/50)=1.5/2.0, = 0.003, =0.05

54 / 58

Massachusetts: Re-Opening

Apr May Jun Jul Aug Sep Oct

2020

0

5

10

15

20

25

30

35

40

45

50

Dai

ly d

eath

s per

mil

lion p

eopleMassachusetts

R0(t)=1.6, R

0(suppress)=2.7, R

0(25/50)=2.0/2.5, = 0.003, =0.20

55 / 58

Washington, DC: Re-Opening

Apr May Jun Jul Aug Sep Oct

2020

0

5

10

15

20

25

Dai

ly d

eath

s per

mil

lion p

eople

District of Columbia

R0(t)=1.3, R

0(suppress)=1.4, R

0(25/50)=1.5/1.6, = 0.003, =0.02

56 / 58

Conclusions

Speculations based on model, we are not epidemiologists

• Time-varying β (or R0) needed to capture social distancing, by

individuals or via policy

• New York City already has around 1700 deaths per million

people

◦ So δ > .002 is required by model

◦ 100% are not already infected and resolved

(continued...)

57 / 58

Conclusions (continued)

• Random sampling in NYC would be very informative about δ and

percent ever infected (for herd immunity effects)

◦ NEJM study of 215 mothers giving birth in NYC found a 15

percent infection rate from Mar 22 to Apr 4

◦ Suggests a high current ever-infected rate (infections

doubling every 3-4 days)

◦ Model suggests δ < .004 to match this

◦ California serology testing also suggests δ ≈ .003 (but

problems?)

• “One size fits all” will not work for re-opening

◦ NYC could potentially re-open soon w/ basic precautions

◦ Rest of NY state could not

58 / 58