Integration of Smalltalk Systems Orlando 2005

1

Eurydike()

SchemalessObject Relational SQL Mapper

Georg Heeg eK12EurydikeEasilyUsable

RelationalYobbishDatabase

IntuitiveKeymappinglessEnvironment

3ObjectsSmalltalk Objects liveObjects respond to messagesObjects have identity

4PersistenceLong timeSharingmultiple usersconcurrent changes

5Objects and PersistenceSave the imagelong termmultiple users have copiesno concurrent changesSerialize ObjectsUse GemStone/SHuge image on diskmulti-usertransaction bases synchronization

6Serialize ObjectsTimestamp now storeString'(Timestamp readFrom: ''06/09/2013 14:38:53.432'' readStream)' Issue: cyclic structures solved by Hubert Baumeister 1988 using keys and IdentityDictionariesBOSSBinary Object Storage SystemParcelsSpecial serializer for codeOthersVOSS, XML, JSON, ODMG

7Store all or nothingNo concurrencyDesire for databasesSerialize Objects8A few Pictures

9Relational databasesFinite sets of typed valuesNumbers, strings, Equalityof valuesCross productscalled tables

10Relations ObjectsTable rows as objectseasy as Objects can describe everytingtables are mapped onto classescolumns are mapped onto instance variablesObjectStudio database supportEXDI (VisualWorks)GemConnectVA Database support11Objects RelationsMuch more interestingMuch more difficultdue to lack of flexibilty on the relation sideTypical solutionOR MappingObjects Networks map onto Entity Relationship Models 12Object Lens'the industry's first Object Relational Mapping (ORM) technology, providing seamless integration between Smalltalk-80 applications and RDBMSs'Richard Steiger Parc, ParcPlace and EnsembleInvented Object Relational Mapping (ORM)Smalltalk Type System, providing strong typing, data entry validation, and program transformation inferencing

13TOPLinkTOPLink is an application framework allows Smalltalk applications to access relational databasesprovides a high degree of integration between the Smalltalk objects and the relational database tablesrelational database can be made to behave like an object database

14GLORP

Generic Lightweight Object-Relational Persistenceopen-sourceobject-relational mapping layer for SmalltalkCamp Smalltalk Projectled byAlan Knightgoal isto provide a simple, powerful framework for reading and writing objects from relational databasesoriginally sponsored by The Object People concepts are reminiscent of TOPLinkthough noticably different in some respects.15PropertiesClass structures map to ER modelsChange is difficultchange classeschange tablesschema migrationdata migrationchange mapping

16Orpheus

Goal: Easy changeNo more database schema migrationsNo more data migrationsDesigned and developed for Component Studio (ERP)Way: universal database schemaone schema for all objectsBased upon Object LensMapping is reduced to typing17CREATE TABLE domainmodel(domainmodelclass character varying(64),objectid integer)Orpheus object table18CREATE TABLE objectreference(attribute character varying(40),basevalue numeric,myinstance integer)Object attribute table19Integer attributes tableCREATE TABLE integervalue(attribute character varying(40),basevalue integer,myinstance integer)20Many more basic type tablesExamplesARBITRARYSTRINGBINARYVALUEBOOLEANVALUELARGESTRINGTINYSTRINGTIMESTAMPVALUE21Our DreamPersistance without painIntuitive for SmalltalkersEasy to use

A Smalltalk as Arden Thomas would say22Can we use the existing ORMs?Let us look at GlorpOrpheusGLORPExplicit description in methods #classModelForXXX:#tableForXXX:#descriptorForXXX:Subclasses need special considerations23OrpheusEvery object class needsa Lens description methodRelationship classes required for N:M relationshipsAll OR-Mappingsrequire typed Smalltalk25Eurydike Wishlist EASYWe want to store any objectWe do not want to write any mapping

26Eurydike Timeline Start of the research project 28 February 2013Project taskBuild schema-less system for storage of any object in a SQL DBSTIC 2013: PresentationSince STIC 2013: Used internally at Heeg Used in GH seaMS Experience starts floats back

27Eurydike Tasks Which tables in the database?How to do the (invisibe) mapping object access to SQL?What about (invisible) proxies?What about concurrency?What about garbage collection in the database?28Research StrategyBottom upDatabase Tables and Mappings firstGarbage Collection lastTry out multiple ideas29Idea ATake the Orpheus Model and make it flexibleOne table per typemakes queries unworkableThusOne table for all the types30Idea A - TableCREATE TABLE fields( objectid integer,rowid integer,

objectclass character varying,

stringfieldkey character varying,stringfieldvalue character varying,

integerfieldkey character varying, integerfieldvalue integer,

referencefieldkey character varying,referencefieldvalue integer)31Eurydike Idea A Example 1association := 'KeyA' -> 1.32objectidrowidownerclassstring fieldkeystring fieldvalueinteger fieldkeyinteger fieldvaluereference fieldkeyreference fieldvalue 11Association'key''KeyA''value'1Eurydike Idea A Example 2association := 'KeyB' -> nil.33objectidrowidownerclassstring fieldkeystring fieldvalueinteger fieldkeyinteger fieldvaluereference fieldkeyreference fieldvalue 11Association'key''KeyA''value'121Association'key''KeyB'Eurydike Idea A Example 3association := 'KeyC' -> 'ValueC'.34objectidrowidownerclassstring fieldkeystring fieldvalueinteger fieldkeyinteger fieldvaluereference fieldkeyreference fieldvalue 11Association'key''KeyA''value'121Association'key''KeyB'31Association'key''KeyC'32Association'value''ValueC'Eurydike Idea A Self Referencesassociation := 'KeyD' -> nil.assocation value: association

association := session detect: [:each | each value = each]35objectidrowidownerclassstring fieldkeystring fieldvalueinteger fieldkeyinteger fieldvaluereference fieldkeyreference fieldvalue 11Association'key''KeyA''value'121Association'key''KeyB'31Association'key''KeyC'32Association'value''ValueC'41Association'key''KeyD''value'4Eurydike Idea A Sequenceablesarray := Array with: 'elementA' with: 200 with: 'elementC' with: 400.

array := session detect: [:each | (each at: 1) = 'elementA']]array := session detect: [:each | (each at: 2) = 200]

The same happens for all other sequenceable collections.

Unordered collections like Set are stored with fictive index as key.36 groupidrowidownerclassstring fieldkeystring fieldvalueinteger fieldkeyinteger field value11Array'1''elementA''2'20012Array'3''elementC''4'400Eurydike Idea A DictionariesDictionariesare collections of associationsare stored as collections of associations37Eurydike Idea A SQL[:base | base name = 'hello']

SELECT fields.groupid 'baseid', embedded.filterid 'filterid'FROMfields,(SELECT base.groupid 'baseid', base.groupid 'filterid'FROM fields base) embeddedWHEREfields.groupid = embedded.filterid ANDfields.stringfieldkey = 'name' ANDfields.stringfieldvalue = 'hello'38Eurydike Idea A - SQLSQL generation is complicated double dispatching for each possible datatype in tableNew expressions need new code paths for all datatypes39Eurydike Idea A - SummaryEasy to useAlmost any Object can be storedOur world is a sequence of 'Object allInstances'we can search with #select:, #detect: and #reject:

Not quite excellent enoughSQL generation is complex because of primitive datatypes and unknown object typeOne column pair per type makes queries complicated

40Idea BTake Idea A and make it simplerThusUnifiy all types as string pairsStore every object as a dictionary41Idea B - TableCREATE TABLE fields( objectid character varying,

keybasetype character varying,keytype character varying,keymajor character varying,keyminor character varying,

valuebasetype character varying,valuetype character varying,valuemajor character varying,valueminor character varying)42Eurydike Idea B#baseType is like a class hierarchy filter for comparing. Only things with same #baseType are compared in SQL queries#type is the real class of object for instantiation#major is the primary value, e.g. the nominator for Number#minor is the secondary value, e.g. the denominator for NumberEurydike Idea B Example 1association := 'KeyA' -> 1.44objectidkeybasetypekeytypeKeymajorkeyminorvaluebasetypevaluetypevaluemajorvalueminor1'Core.Symbol''Core.ByteSymbol''class''Core.CharacterArray''Core.ByteString''Core.Association'1'Core.Symbol''Core.ByteSymbol''key''Core.CharacterArray'Core.ByteString''KeyA'1'Core.Symbol''Core.ByteSymbol''value''Core.Number''Core.SmallInteger''1''1'Eurydike Idea B Example 2association := 'KeyB' -> nil.45objectidkeybasetypekeytypekeymajorkeyminorvaluebasetypevaluetypevaluemajorvalueminor1'Core.Symbol''Core.ByteSymbol''class''Core.CharacterArray''Core.ByteString''Core.Association'1'Core.Symbol''Core.ByteSymbol''key''Core.CharacterArray'Core.ByteString''KeyA'1'Core.Symbol''Core.ByteSymbol''value''Core.Number''Core.SmallInteger''1''1'2'Core.Symbol''Core.ByteSymbol''class''Core.CharacterArray''Core.ByteString''Core.Association'2'Core.Symbol''Core.ByteSymbol''key''Core.CharacterArray'Core.ByteString''KeyB'Eurydike Idea B Example 3association := 'KeyC' -> 'ValueC'.46objectidkeybasetypekeytypekeymajorkeyminorvaluebasetypevaluetypevaluemajorvalueminor1'Core.Symbol''Core.ByteSymbol''class''Core.CharacterArray''Core.ByteString''Core.Association'1'Core.Symbol''Core.ByteSymbol''key''Core.CharacterArray'Core.ByteString''KeyA'1'Core.Symbol''Core.ByteSymbol''value''Core.Number''Core.SmallInteger''1''1'2'Core.Symbol''Core.ByteSymbol''class''Core.CharacterArray''Core.ByteString''Core.Association'2'Core.Symbol''Core.ByteSymbol''key''Core.CharacterArray'Core.ByteString''KeyB'3'Core.Symbol''Core.ByteSymbol''class''Core.CharacterArray''Core.ByteString''Core.Association'3'Core.Symbol''Core.ByteSymbol''key''Core.CharacterArray'Core.ByteString''KeyC'3'Core.Symbol''Core.ByteSymbol''value''Core.CharacterArray'Core.ByteString''ValueC'Eurydike Idea B Self Referencesassociation := 'KeyD' -> nil.assocation value: associationassociation := session detect: [:each | each value = each]47objectidkeybasetypekeytypekeymajorkeyminorvaluebasetypevaluetypevaluemajorvalueminor1'Core.Symbol''Core.ByteSymbol''class''Core.CharacterArray''Core.ByteString''Core.Association'1'Core.Symbol''Core.ByteSymbol''key''Core.CharacterArray'Core.ByteString''KeyA'1'Core.Symbol''Core.ByteSymbol''value''Core.Number''Core.SmallInteger''1''1'2'Core.Symbol''Core.ByteSymbol''class''Core.CharacterArray''Core.ByteString''Core.Association'2'Core.Symbol''Core.ByteSymbol''key''Core.CharacterArray'Core.ByteString''KeyB'3'Core.Symbol''Core.ByteSymbol''class''Core.CharacterArray''Core.ByteString''Core.Association'3'Core.Symbol''Core.ByteSymbol''key''Core.CharacterArray'Core.ByteString''KeyC'3'Core.Symbol''Core.ByteSymbol''value''Core.CharacterArray'Core.ByteString''ValueC'4'Core.Symbol''Core.ByteSymbol''class''Core.CharacterArray''Core.ByteString''Core.Association'4'Core.Symbol''Core.ByteSymbol''key''Core.CharacterArray'Core.ByteString''KeyD'4'Core.Symbol''Core.ByteSymbol''value''Core.Object''Core.Object''4'Eurydike Idea B - DictionariesIn Idea A Dictionaries are collections of AssociationsIn Idea B there is native implemantation of Dictionariesd := Dictionary newd at: 'KeyA' put: 1.d at: 'KeyC' put: 'ValueC'objectidkeybasetypekeytypekeymajorkeyminorvaluebasetypevaluetypevaluemajorvalueminor5'Core.Symbol''Core.ByteSymbol''class''Core.CharacterArray''Core.ByteString''Core.Dictionary'5'Core.CharacterArray'Core.ByteString''KeyA''Core.Number''Core.SmallInteger''1''1'5'Core.CharacterArray''Core.ByteString''KeyC''Core.CharacterArray''Core.ByteString''ValueC'As we store dictionaries we need only expressions for the primitive dictionary access like:

#at:, #keyAtValue:, #keys, #values, #size, #select:

The rest can be compound by these primitives.

includes: aValue^self values select: [:each | each = aValue]

includesKey: aKey^self keys select: [:each | each = aKey]49Eurydike Idea B - ExpressionsAs we return uniform dataleads to simpler SQL-generationno need for context informationSpecial expression combinations can be optimized later when make it fast gets importantSpecial ObjectsByteArraysBase 64 encoded stringsCompiled Methodsas source code oras byte array of compiled codeBlockContextssame consirations needed as for OpenTalk STST51Eurydike Idea B - SummarySame advantages as Idea ASimple SQL generationExpressions can be compound by primitivesUniform data -> easier expression and SQL generationUniform data -> different Numbers like double, fraction, integer can be compared in queriesMost of the collection protocol is possible like #includes:, #includesKey:, #anySatisfy:, #allSatisfy:

ButEurydike - ObjectSpaceIt is not the Smalltalk-way to store and retrieve big unstructured collections of instances Smalltalk wants a root objects and navigates from object to objectEurydike implements abstract class ObjectSpaceObjectSpace Example create a class for the objects

Smalltalk.Eurydike defineClass: #Personsuperclass: #{Core.Object}instanceVariableNames: 'name lastName address '

One ObjectSpace SubclassSmalltalk.Eurydike defineClass: #PersonSpacesuperclass: #{Eurydike.ObjectSpace}instanceVariableNames: 'persons '55One instance methodinitialize

super initialize.persons := OrderedCollection new56One class methodconnectionProfile

^(ConnectionProfile new)platform: PostgreSQLPlatform new;environment: 'localhost:5432_postgres';username: 'postgres';password: 'postgres';yourself57Create the objectsp := PersonSpace default.p persons add: ((Person new)name: 'Jrg'; lastName: 'Belger'; address: 'Kthen').p persons add: ((Person new)name: 'Karsten'; lastName: 'Kusche'; address: 'Kthen').p persons add: ((Person new)name: 'Magnus'; lastName: 'Schwarz'; address: 'Dortmund').p commit58PGAdmin shows the result59

Eurydike Schema migrationp := PersonSpace default.d := Dictionary new.p persons do: [:person | d at: person lastName put: person].p persons: d.p commitResult61

Getting rid of garbagePersonSpace default garbageCollect62Cleaned up database63

Incremental GarbageCollectDELETE FROM fields WHERE fields.id NOT IN ( SELECT DISTINCT reffields.id FROM fields "reffields", fields WHERE (reffields.keymajor = 'class' AND reffields.valuemajor = 'Eurydike.PersonSpace') OR (fields.keybasetype = 'Core.Object' ANDfields.keymajor = reffields.id) OR (fields.valuebasetype = 'Core.Object' ANDfields.valuemajor = reffields.id))64Global GarbageCollectDELETE FROM fields WHERE fields.id NOT IN ((WITH RECURSIVE refs(id) AS ( SELECT fields.id FROM fields WHERE fields.keymajor = 'class' AND fields.valuemajor = 'Eurydike.PersonSpace' UNION ALL SELECT fields.valuemajor FROM fields, refs WHERE fields.valuebasetype = 'Core.Object' AND fields.id = refs.id ) SELECT * FROM refs)

UNION ALL

(WITH RECURSIVE refs(id) AS ( SELECT fields.id FROM fields WHERE fields.keymajor = 'class' AND fields.valuemajor = 'Eurydike.PersonSpace' UNION ALL SELECT fields.keymajor FROM fields, refs WHERE fields.keybasetype = 'Core.Object' AND fields.id = refs.id ) SELECT * FROM refs))65Eurydike Current StatusUsed in GH seaMSnew content management system based on seaBreeze, seaBreeze Mobile and Seasidecurrently in initial use internallyplanned to be available on Cincom Marketplace when released66Eurydike Future Multi User Life connectionsGlobalLocksemaphore like objectSELECT FOR UPDATEsingle object lock

ongoing research67Market of EurydikeEasy object based persistancy in medium size projects68

69Our Team made the dream come true

Georg Heeg eKGeorg Heeg eKBaroper Str. 33744227 DortmundGermanyTel: +49 (0)231 - 97599 - 0Fax: +49 (0)231 - 97599-20Email: georg@heeg.dehttp://www.heeg.deGeorg Heeg AGSeestrae 1358027 ZrichSwitzerlandTel: +41 (848) 43 34 24Georg Heeg eKWallstrae 2206366 KthenGermanyTel: +49 (0)3496 - 214 328Fax: +49 (0)3496 - 214 712