10/18/17

1

AssessingCriticalThinkingSkillsDAVIDK.KNOX,PH.D.,DIRECTOROFCLEMSONTHINKS2(CT2)

κριτικός (Kritikos), relatingtojudging,fitforjudging,skilledinjudging,abletodiscern (Plato, Plutarch, Lucian,others).Withthegenitiveoftheobject,forexample:κριτικὸς ἐνθυμήσεων καί ἐννοιῶν καρδίας- abletojudgethe thoughtsandintentions oftheheart.

Whatis”Critical”inCriticalThinking?

Attitude+Knowledge+ThinkingSkills=CriticalThinking

“Proficiencyinreading,writing,andarithmetichastraditionallybeentheentry-levelthresholdtothejobmarket,butthenewworkplacerequiresmorefromitsemployees.Employeesneedtothinkcritically,solveproblems,innovate,collaborate,andcommunicatemoreeffectively.”–– AmericanManagementAssociation,2010(quotedinDianeHalpern:ThoughtandKnowledge,page5.)

10/18/17

2

HartResearchAssociatesStudy:2009*• Intellectualandpracticalskills• Theabilitytocommunicateeffectively,orallyandinwriting(89%)• Criticalthinkingandanalyticalreasoningskills(81%)• Theabilitytoanalyzeandsolvecomplexproblems(75%)• Teamworkskillsandtheabilitytocollaboratewithothersindiversegroupsettings(71%)

• Theabilitytoinnovateandbecreative(70%)• Theabilitytolocate,organize,andevaluateinformationfrommultiple• sources(68%)• Theabilitytoworkwithnumbersandunderstandstatistics(63%)

*http://www.aacu.org/sites/default/files/files/LEAP/2009_EmployerSurvey.pdf

HartResearchAssociatesStudy:2009*

*http://www.aacu.org/sites/default/files/files/LEAP/2009_EmployerSurvey.pdf

CriticalThinking,Defined“Criticalthinkingistheprocessofpurposeful,reflective

judgment.Criticalthinkingmanifestsitselfingivingreasonedandfair-mindedconsiderationtoevidence,conceptualizations,methods,contexts,andstandards inordertodecidewhatto

believeorwhattodo.”(Facione,2010).“Theintellectuallydisciplinedprocessofactively andskillfullyconceptualizing,applying,analyzing,synthesizing,and/orevaluatinginformationgatheredfrom,orgeneratedby,

observation,experience,reflection,reasoning,orcommunication,asaguidetobeliefandaction”(ScrivenandPaul,1987).

10/18/17

3

CriticalThinking,Defined“Criticalthinkingistheuseofthosecognitiveskillsorstrategiesthatincreasetheprobabilityofadesiredoutcome.Itisusedtodescribethinkingthatispurposeful,reasoned,andgoal

directed– thekindofthinkinginvolvedinsolvingproblems,formulatinginferences,calculatinglikelihoods,andmakingdecisions,whenthethinkerisusingskillsthatare

thoughtfulandeffectivefortheparticularcontextandtypeofthinkingtask.”(Halpern,2014)

“Beingacriticalthinkerinvolvesmorethancognitiveactivitiessuchaslogicalreasoningorscrutinizingargumentsforassertionsunsupportedbyempiricalevidence.Thinkingcriticallyinvolvesourrecognizingtheassumptionsunderlyingourbeliefsandbehaviors.Itmeanswecangivejustificationsforourideasandactions.Mostimportant,perhaps,itmeanswetrytojudgetherationalityofthesejustifications.Wecandothisbycomparingthemtoarangeofvaryinginterpretationsandperspectives.Wecanthinkthrough,project,andanticipatetheconsequencesofthoseactionsthatarebasedonthesejustifications.Andwecantesttheaccuracyandrationalityofthesejustificationsagainstsomekindofobjectiveanalysisofthe

"real"worldasweunderstandit.”(Brookfield,1987)

CriticalThinking,Defined“Itisthemarkofaneducatedmindtobeabletoentertainathoughtwithoutacceptingit.”―Aristotle,Metaphysics“Allthatweareistheresultofwhatwehavethought:itisfoundedonourthoughtsandmadeupofourthoughts.”–– Buddha,Dhammapada“Learningwithoutthoughtislaborlost;thoughtwithoutlearningisperilous”–– Confucius,Analects“Readnottocontradictandconfute;nortobelieveandtakeforgranted;nortofindtalkanddiscourse;buttoweighandconsider”–– FrancisBacon,“OfStudies”

CriticalThinkingMindset:EagernessandSkills“Apersonwithastrongdisposition towardcriticalthinkinghastheconsistentinternalmotivationtoengageproblemsandmakedecisionsbycriticalthinking.Operationallythismeansthreethings:Thepersonconsistentlyvalues criticalthinking,believes thatusingcriticalthinkingskillsoffersthegreatestpromiseforreachinggoodjudgments,andintends toapproachproblemsanddecisionsbyapplyingcriticalthinkingskillsasbestasheorshecan.Thiscombinationofvalues,beliefs,andintentionsformsthehabitsofmindthatdisposethepersontowardcriticalthinking.”–– PeterFacione,ThinkCritically,Chapter2.1

10/18/17

4

CriticalThinkingMindset:EagernessandSkills“Everyonethinks;itisournaturetodoso.Butmuchofourthinking,lefttoitself,isbiased,distorted,partial,uninformed,ordown-rightprejudiced.Yetthequalityofourlifeandthatofwhatweproduce,make,orbuilddependspreciselyonthequalityofourthought.Shoddythinkingiscostly,bothinmoneyandinqualityoflife.Excellenceinthought,however,mustbesystematicallycultivated”–– RichardPaulandLindaElder,CriticalThinkingConceptsandTools,pg.2

CriticalThinkingMindset:EagernessandSkillsUniversalIntellectualStandards:Clarity: Couldyouelaboratefurtheronthatpoint?Couldyouexpressthatpointinanotherway?Couldyougivemeanillustration?”Couldyougivemeanexample?Accuracy: Isthatreallytrue?Howcouldwecheckthat?Howcouldwefindoutifthatistrue?Precision: Couldyougivememoredetails?Couldyoubemorespecific?Relevance: Howisthatconnectedtothequestion?Howdoesthatbearontheissue?

CriticalThinkingMindset:EagernessandSkillsUniversalIntellectualStandards(cont’d)

Depth: Howdoesyouransweraddressthecomplexitiesinthequestion?Howareyoutakingintoaccounttheproblemsinthequestion?Areyoudealingwiththemostsignificantfactors?Breadth: Doweneedtoconsideranotherpointofview?Isthereanotherwaytolookatthisquestion?WhatwouldthislooklikefromthepointofviewofX?Logic: Doesthisreallymakesense?Doesthisfollowfromwhatyousaid?Howdoesthatfollow?Fairness: Areweconsideringallrelevantviewpointsingoodfaith?Arewedistortingsomeinformationtomaintainourbiasedperspective?Arewemoreinterestedinourvestedintereststhanthecommongood?–– RichardPaulandLindaElder,CriticalThinkingConceptsandTools,pgs.8,9

10/18/17

5

CT2 PointsofAssessment:Classes•PreandPostCaliforniaCriticalThinkingSkillsTests

•CTArtifacts

•AdditionalquestionsforCT2 classesonEvaluationofInstructorforms

•TransferrableSkillsNSSEmodule

•ETSProficiencyProfile(FreshmenandSeniors)

•ClemsonEducationalProfile

CT2 PointsofAssessment:

CriticalThinkingTestsHigherEducation:General-Content,Multi-Aspect*ACTCAAPTestModule:CriticalThinking. .http://www.act.org/caap/index.html (retiringinJanuary2018)

TheCaliforniaCriticalThinkingSkillsTest(CCTST): CollegeLevel.http://www.insightassessment.com

TheCaliforniaCriticalThinkingDispositionsInventory (CCTDI). http://www.insightassessment.com

CambridgeThinkingSkillsAssessment.CambridgeAssessmenthttp://www.admissionstests.cambridgeassessment.org.uk/adt/tsacambridge/about

CollegiateLearningAssessment+ (CLA+).http://www.collegiatelearningassessment.org/

TheCriticalThinkingAssessmentTest (CAT).http://www.tntech.edu/cat/using/CornellCriticalThinkingTest,LevelZ.https://www.criticalthinking.com/cornell-critical-thinking-test-level-z.html

ETS® ProficiencyProfile. http://www.ets.org/proficiencyprofile/about

*Adaptedfrom:ANANNOTATEDLISTOFENGLISH-LANGUAGECRITICALTHINKINGTESTS: RobertH.EnnisandGeoffreyScottChattin,UniversityofIllinoisUC

10/18/17

6

CriticalThinkingTestsHigherEducation:General-Content,Multi-Aspect*HalpernCriticalThinkingAssessment(HCTA)https://sites.google.com/site/dianehalperncmc/home/research/halpern-critical-thinking-assessment

HolisticCriticalThinkingScoringRubric(HCTSR).http://www.insightassessment.com/

InternationalCriticalThinkingBasicConcepts&UnderstandingOnlineTest:AssessingInitialUnderstandingofBasicCriticalThinkingConceptsandPrinciples.http://www.criticalthinking.org/pages/online-critical-thinking-basic-concepts-test/679

InternationalCriticalThinkingEssayTest.http://www.criticalthinking.org/store/products/international-critical-thinking-test/185

JamesMadisonTestofCriticalThinking(partoftheJamesMadisoncriticalthinkingcourse),https://www.criticalthinking.com/james-madison-critical-thinking-course-student-book.html.

NewJerseyTestofReasoningSkills.http://cehs.montclair.edu/academic/iapc/

OxfordThinkingSkillsAssessment.CambridgeAssessment.http://www.admissionstests.cambridgeassessment.org.uk/adt/tsaoxford/about

Watson-GlaserIICriticalThinkingAppraisal.https://us.talentlens.com/watson-glaser-critical-thinking-test

*Adaptedfrom:ANANNOTATEDLISTOFENGLISH-LANGUAGECRITICALTHINKINGTESTS: RobertH.EnnisandGeoffreyScottChattin,UniversityofIllinoisUC

CriticalThinkingTestsTheCaliforniaCriticalThinkingSkillsTest(CCTST)TheCaliforniaCriticalThinkingSkillsTest(CCTST)isanobjectivemeasureofthecorereasoningskillsneededforreflectivedecisionmakingconcerningwhattobelieveorwhattodo.TheCCTSTisdesignedtoengagethetest-taker’sreasoningskills.Multiplechoiceitemsuseeverydayscenarios,appropriatetotheintendedtest-takergroup.Eachitemrequiresthatthetest-takermakeanaccurateandcompleteinterpretationofthequestion. Anyspecializedinformationneededtorespondcorrectlyisprovidedinthequestionitself.Thetestitemsrangeindifficultyandcomplexity.Differentquestionsprogressivelyinvitetest-takerstoanalyzeortointerpretinformationpresentedintext,charts,orimages;todrawaccurateandwarrantedinferences;toevaluateinferencesandexplainwhytheyrepresentstrongreasoningorweakreasoning;ortoexplainwhyagivenevaluationofaninferenceisstrongorweak.Theinstrumentistypicallyadministeredin45-50minutes;thelengthoftheinstrumentissettopermitmaximumperformancewithintherangeofpossibleeffortfortheintendedtest-takergroup.

CriticalThinkingTestsTheCaliforniaCriticalThinkingSkillsTest(CCTST)ScaleDescriptions

ReasoningSkills- Overall:

TheReasoningSkillsOveralldescribesoverallstrengthinusingreasoningtoformreflectivejudgmentsaboutwhattobelieve or whattodo.Toscorewelloverall,thetest-takermustexcelinthesustained,focusedandintegratedapplicationofcorereasoningskillsincludinganalysis,interpretation,inference,evaluation,explanation,inductionanddeduction.The Overallscorepredictsthecapacityforsuccessineducationalorworkplacesettingswhichdemandreasoneddecisionmakingandthoughtfulproblemsolving.

Analysis:

Analyticalreasoningskillsenablepeopletoidentifyassumptions,reasonsandclaims,andtoexaminehowtheyinteractintheformationofarguments. Weuseanalysistogatherinformationfromcharts,graphs,diagrams,spokenlanguageanddocuments. Peoplewithstronganalyticalskillsattendto patternsandtodetails.Theyidentifytheelementsofasituationanddeterminehowthosepartsinteract. Stronginterpretationskillscansupporthighqualityanalysisbyprovidinginsightsintothesignificanceofwhatapersonissayingorwhatsomethingmeans.

Inference:

Inferenceskillsenableustodrawconclusionsfromreasonsandevidence.Weuseinferencewhenweofferthoughtfulsuggestionsandhypotheses. Inferenceskillsindicatethenecessaryortheveryprobableconsequencesofagivensetoffactsandconditions. Conclusions,hypotheses,recommendationsordecisionsthatarebasedonfaultyanalyses,misinformation,baddataorbiasedevaluationscanturnouttobemistaken,eveniftheyhavebeenreachedusingexcellentinferenceskills.

10/18/17

7

CriticalThinkingTestsTheCaliforniaCriticalThinkingSkillsTest(CCTST)ScaleDescriptions,Continued

Evaluation:

Evaluativereasoningskillsenableustoassessthecredibilityofsourcesofinformationandtheclaimstheymake. And,weuse theseskillstodeterminethestrengthorweaknessofarguments. Applyingevaluationskillswecanjudgethequalityofanalyses,interpretations,explanations,inferences,options,opinions,beliefs,ideas,proposals,anddecisions. Strongexplanationskillscansupporthighqualityevaluationbyprovidingtheevidence,reasons,methods,criteria,orassumptionsbehindtheclaimsmadeandtheconclusionsreached.

Deduction:

Decisionmakinginpreciselydefinedcontextswhererules,operatingconditions,corebeliefs,values,policies,principles,proceduresandterminologycompletelydeterminetheoutcomedependsonstrongdeductivereasoningskills.Deductivereasoningmoveswithexactingprecisionfromtheassumedtruthofasetofbeliefstoaconclusionwhichcannotbefalseifthosebeliefsaretrue.Deductivevalidityisrigorouslylogicalandclear-cut. Deductivevalidityleavesno roomforuncertainty,unlessonealtersthemeaningsofwordsorthegrammarofthelanguage.

Induction:

Decisionmakingincontextsofuncertaintyreliesoninductivereasoning. Weuseinductivereasoningskillswhenwedrawinferencesaboutwhatwethinkmustprobablybetruebasedonanalogies,casestudies,priorexperience,statisticalanalyses,simulations,hypotheticals,andfamiliarcircumstancesandpatternsofbehavior. Aslongasthereisthepossibility,howeverremote,thatahighlyprobableconclusionmightbemistaken,thereasoningisinductive.Althoughit doesnotyieldcertainty,inductivereasoningcanprovideasolidbasisforconfidenceinourconclusions.

10/18/17

8

Pre-Test

Post-Test

CoreCriticalThinkingSkillsMeasuredbytheCCTST

CriticalThinkingTestsTheCriticalThinkingAssessmentTest(CAT)TheCATinstrument,developedatTennesseeTech,isauniquetooldesignedtoassessandpromotetheimprovementofcriticalthinkingandreal-worldproblemsolvingskills.Theinstrumentistheproductofextensivedevelopment,testing,andrefinementwithabroadrangeofinstitutions,faculty,andstudentsacrossthecountry.TheNationalScienceFoundationhasprovidedsupportformanyoftheseactivities.TheCATinstrumentisdesignedtoassessabroadrangeofskillsthatfacultyacrossthecountryfeelareimportantcomponentsofcriticalthinkingandrealworldproblemsolving.Thetestwasdesignedtobeinterestingandengagingforstudents.Allofthequestionsarederivedfromrealworldsituations.Mostofthequestionsrequireshortansweressayresponses,andadetailedscoringguidehelpsensuregoodscoringreliability.

10/18/17

9

CriticalThinkingTestsTheCriticalThinkingAssessmentTest(CAT)

SkillsAssessedbyCATInstrument

EvaluatingInformation

•Separatefactualinformationfrominferences.

•Interpretnumericalrelationshipsingraphs.

•Understandthelimitationsofcorrelationaldata.

•Evaluateevidenceandidentifyinappropriateconclusions.

CreativeThinking

•Identifyalternativeinterpretationsfordataorobservations.

•Identifynewinformationthatmightsupportor

contradictahypothesis.

•Explainhownewinformationcanchangeaproblem.

LearningandProblemSolving

•Separaterelevantfromirrelevantinformation.

•Integrateinformationtosolveproblems.

•Learnandapplynewinformation.

•Usemathematicalskillstosolvereal-worldproblems.

•Communication

•Communicateideaseffectively.

CriticalThinkingTestsResponseInteractingwithpeoplefromacrosscampusInteractionwithcolleaguesandfellowfacultymembers.Thecollegialityandopportunitytoworkwithpeoplefromoutsideofmydiscipline.Theconversationswehadatourtable,bothformalandinformal.InteractingwithotherfacultyfromthisgroupTheinteractionwiththeotherfacultymembersmadethescoringsessionsquiteenjoyable.Thecamaraderieandtheintellectualdiscussions

Thediscussionsatthetableaboutthescoring,etc.Talkingabouthowtoapplytherubricinparticularcaseswas,attimes,eye-opening.

CollegialityamongalltheparticipantsTheopportunitytoexchangeideaswithothercolleagues.Meetingotherfacultyfromotherdepartments.Interactionwithotherlike-mindedCT2faculty.InteractionwithfacultywhilesharingacommongoalThecamaraderieandthequestions.interactionwithfacultyfromvariousdisciplinesEngagingindialoguetoreachconsensuswithcolleagues

Q.WhatdidyoulikemostabouttheCATTestScoringSession?

CriticalThinkingTests

10/18/17

10

Assessments:CCTST vs.CAT• CaliforniaCriticalThinkingSkillsTest(CCTST)• 34multiplechoicequestions• 45minutes• Onlineorpaper• Gradedimmediately(online)• Multi-LingualVersions• $10pertest

**Studentsaretestedduringthefirstandfinalweeksofclasses

• CriticalThinkingAssessmentTest(CAT)• Essay:15questions• Onehour• Paperandpencil• Facultygraded• Facultydevelopmentpotential• $9.95pertestand$300annualfee

Assessments:ETS®ProficiencyProfileAstheonlyintegratedtestofgeneraleducationskills,theETSProficiencytestassessesfourcoreskillareas- criticalthinking,reading,writingandmathematics- inasingletestthatthe VoluntarySystemofAccountability(VSA)hasselectedasagaugeofgeneraleducationoutcomes.

ClemsonUniversity usesthe ETSProficiencyProfile testto:• Functionas“bookends”intheCT2 assessmentprocess• measureanddocumentprogrameffectivenesstomeetrequirementsforaccreditationandprogramfunding• assessstudentproficiencyincoreacademicskillareastoidentifystrengths,weaknessesandopportunitiestoimprovecurriculumand

instruction• compareperformanceagainstprogramsatmorethan380institutionsnationwide• conductbenchmarkandtrendanalysestoevaluateprogramimprovementeffortsandoveralllearningoutcomes

ClemsonUniversitybeganinFall2011toadministertheETSProficiencyProfiletoallincomingfreshmen.Thesesamestudentsaretestedwhentheyareseniorsandwewillbeabletolongitudinallytracktheirprogress.Thisprovidesuswithalevelofinsightintothe“value-added” aspectofaClemsoneducationneverpossiblebefore.

Assessments:ETS®ProficiencyProfileETSProficiencyProfileStandardForm

TheStandardformoftheETSProficiencyProfiletestisintendedtoprovideinformationaboutindividualstudentsaswellasgroupsofstudents.Itconsistsof108questions,dividedintotwosectionsof54questionseach.Thetwosectionsmaybeadministeredeitherinasingletwo-hourtestingsessionorintwoseparate,one-hourtestingsessions.TheStandardformincludes:• 27questionstestingcriticalthinkingskills• 27questionstestingreadingskills• 27questionstestingwritingskills• 27questionstestingmathematicsskills

Additionally,studentstakingtheStandardformcanearna CertificateofAchievement basedonhowwelltheyperform.

10/18/17

11

Assessments:ETS®ProficiencyProfileETSProficiencyProfileAbbreviatedForm

TheAbbreviatedFormoftheETSProficiencyProfiletestisintendedtoprovideinformationaboutgroupsofatleast30students.Itisadministeredinasingle40-minutetestingsession.TheAbbreviatedFormdoesnotprovideinformationaboutindividualstudents.

TheAbbreviatedformiscreatedbydividingtheStandardFormintothreesmallerformsof36questionseach.ThethreeAbbreviatedFormsarepackagedinalternatingsequence,sothateachversionistakenbyone-thirdofthestudents.The108questionsintheStandardFormareassignedtothethreeAbbreviatedFormstomakeeachofthemaminiatureversionoftheStandardForm.

EachofthethreeAbbreviatedFormsincludes:• Ninequestionsmeasuringcriticalthinkingskills• Ninequestionsmeasuringreadingskills• Ninequestionsmeasuringwritingskills• Ninequestionsmeasuringmathematicsskills

Artifacts• Defined:ar·ti·fact noun \ˈär-ti-ˌfakt\ a: somethingcreatedbyhumansusuallyforapracticalpurpose;especially : anobjectremainingfromaparticularperiod.

• Eachcoursesubmitsatleastonetypeofartifact

• Cansubmitdifferentartifacts

• Manyclasseshaveaspecificassignmentthatwillbe

designatedthecriticalthinkingartifact.

• Randomsamplefromclassisscored

• ScoringisperformedbyCT2 FacultyScholarsusingthe

AAC&UCriticalThinkingrubric

• Examples:• Essay• Project• Critique• Video• PosterPresentation• Portfolio

Artifacts:AAC&UCriticalThinkingVALUERubric

10/18/17

12

Artifacts:“VisualEssay”

ThankYou!Pleasevisitourwebsiteforfurtherinformation:

http://www.clemson.edu/thinks2

2016ClemsonThinks2 FacultyInstitute