Tracking your data across the fourth dimension

Tracking your data across the fourth dimension

• 1.
  Confoo 2015 Tracking yourdata across the fourth dimension Jeremy Cook
• 3.
  This talk isabout temporal databases
• 4.
  –Wikipedia “A temporal databaseis a database with built-in support for handling data involving time…”
• 5.
  – Jeff Carouth,https://twitter.com/jcarouth/status/496842218674470912 “Tonight @JCook21 explained temporal databases and I’m sure my brain is now leaking out of my nose.”
• 6.
  What problem dotemporal databases solve?
• 7.
  Databases are goodat ‘now’ ❖ Create ❖ Read ❖ Update ❖ Delete ❖ At any point we only see the current state of the data
• 8.
  Databases are goodat ‘now’ ❖ How many people work in each department of the company? ❖ For each product category how many products are in stock? Where is the stock located at? ❖ How many orders are currently in each fulfilment state?
• 9.
  The fourth dimension ❖Show me how salaries paid have changed by department for each quarter over the last 4 years and how they’re forecast to change next year ❖ Show me how stock levels have changed over time. How much stock are we forecast to have at any point in the future? ❖ For audit purposes show me a complete history of every change to this data, what period of time each change was valid for and when we knew about any changes
• 10.
  The fourth dimension
• 11.
  Some Temporal DatabaseTheory
• 12.
  Temporal aspects
• 13.
  Decision Time ❖ Recordsthe time at which a decision was made ❖ Modelled as a single value ❖ Allows for granularity through the data type used
• 14.
  Decision Time EmpId NameHire Date Decision to Hire 1 Jeremy 2014-03-03 2014-01-20 2 Anna 2015-01-02 2013-12-15 3 Yann 2013-08-20 2013-08-20
• 15.
  Valid Time “In temporaldatabases, valid time (VT) is the time period during which a database fact is valid in the modelled reality.” –Wikipedia
• 16.
  Valid Time ❖ Modelledas a period of time between two dates ❖ Lower bound is always closed but upper bound can be open
• 17.
  Valid Time EmpId NameHire date Termination date 1 Jeremy 2014-03-03 2015-01-20 2 Anna 2015-01-02 ∞ 3 Yann 2013-08-20 2015-12-22 4 Colin 2015-05-01 ∞
• 18.
  Valid Time EmpId NameDept Hire date Term date StartVT EndVT 1 Jeremy Dev 2014-03-03 ∞ 2014-03-03 2014-07-30 1 Jeremy QA 2014-03-03 2015-01-20 2015-01-21 2015-01-20 2 Anna Dev 2015-01-02 ∞ 2015-01-02 2015-01-30 2 Anna Mgmt 2015-01-02 ∞ 2015-01-31 ∞ 3 Yann Mgmt 2013-08-20 2015-12-22 2013-08-20 ∞ 4 Colin Dev 2015-05-01 ∞ 2015-05-01 ∞
• 19.
  Job done?
• 20.
  Valid-time on itsown may not be enough! Name Type StartVT EndVT Saturn Planet Billions of years ago ∞ Pluto Planet Billions of years ago ∞
• 21.
  Valid-time on itsown may not be enough! Name Type StartVT EndVT Saturn Planet Billions of years ago ∞ Pluto Dwarf planet Billions of years ago ∞
• 22.
  Valid-time on itsown may not be enough! Name Type StartVT EndVT Saturn Planet Billions of years ago ∞ Pluto Plutoid Billions of years ago ∞
• 23.
  Valid-time on itsown may not be enough! Name Type StartVT EndVT Saturn Planet Billions of years ago ∞ Pluto Planet Billions of years ago 2006 Pluto Dwarf planet 2006 2008 Pluto Plutoid 2008 ∞
• 24.
  Transaction Time “In temporaldatabases, transaction time (TT) is the time period during which a fact stored in the database is considered to be true.” –Wikipedia
• 25.
  Transaction Time ❖ Modelledas a period of time between two dates ❖ Lower bound is always closed but upper bound can be open
• 26.
  Transaction Time Name TypeStartVT EndVT StartTT EndTT Pluto Planet Billions of years ago ∞ 1930 2006 Pluto Dwarf planet Billions of years ago ∞ 2006 2008 Pluto Plutoid Billions of years ago ∞ 2008 ∞
• 27.
  Valid Time !=Transaction Time Name Clothing StartVT EndVT StartTT EndTT Father Christmas null A long time ago ∞ 1973 1975 Santa Claus red A long time ago ∞ 1975 1980 Saint Nicholas red 270 AD ∞ 1980 1982
• 28.
  How many temporalaspects should you use? ❖ As many or few as your application needs! ❖ Tables that implement two aspects are bi-temporal ❖ You can implement more aspects, in which case you have multi temporal tables
• 29.
  Is your headspinning? ❖ Decision time records when a decision was taken ❖ Valid Time records the period of time for which the fact is valid ❖ Transaction Time records the period of time for which the fact is considered to be true
• 30.
  SQL:2011 Temporal
• 31.
  A note onthe example tables CREATE TABLE dept ( DNo INTEGER, DName VARCHAR(255) ); CREATE TABLE emp ( ENo INTEGER, EName VARCHAR(255), EDept INTEGER );
• 32.
  Periods ❖ Table component,capturing a pair of columns defining a start and end date ❖ Not a new data type, but metadata about columns in the table ❖ Closed-open constraint ❖ Enforces that end time > start time
• 33.
  Valid time ❖ Alsocalled application time in SQL:2011 ❖ Modelled as a pair of date time columns with a period ❖ Name of the columns and period is up to you
• 34.
  Valid time ALTER TABLEemp ADD ( EStart DATE, EEnd DATE, PERIOD FOR EPeriod (EStart, EEnd) );
• 35.
  Temporal primary keys ❖SQL:2011 allows a valid time period to be named as part of a primary key ❖ Can also enforce that the valid time periods do not overlap
• 36.
  Temporal primary keys ALTERTABLE emp ADD PRIMARY KEY (ENo, EPeriod);
• 37.
  Temporal primary keys ALTERTABLE emp ADD PRIMARY KEY (ENo, EPeriod WITHOUT OVERLAPS);
• 38.
  Temporal foreign keys ❖What happens if a parent and child table both define valid time periods? ❖ It doesn’t make sense to allow a row in a child table to reference a row in a parent table where the valid time does not overlap ❖ SQL:2011 allows valid time periods to be part of foreign key constraints
• 39.
  Temporal foreign keys ALTERTABLE dept ADD ( DStart DATE, DEnd DATE, PERIOD FOR DPeriod (DStart, DEnd) ); ALTER TABLE emp ADD FOREIGN KEY (Edept, EPeriod) REFERENCES dept (DNo, PERIOD DPeriod);
• 40.
  Querying valid timetables ❖ Can query against valid time columns as normal - they’re just normal table columns ❖ Updates and deletes can be performed for a period of a valid time time period
• 41.
  Querying valid timetables ❖ SQL:2011 allows you to create periods to use in your queries and use new predicates: ❖ CONTAINS ❖ OVERLAPS ❖ EQUALS ❖ PRECEDES ❖ SUCCEEDS ❖ IMMEDIATELY SUCCEEDS and IMMEDIATELY PRECEDES
• 42.
  Querying valid timetables UPDATE Emp FOR PORTION OF EPeriod FROM DATE '2011-02-03' TO DATE '2011-09-10' SET EDept = 4 WHERE ENo = 22217;
• 43.
  Querying valid timetables DELETE Emp FOR PORTION OF EPeriod FROM DATE '2011-02-03' TO DATE '2011-09-10' WHERE ENo = 22217;
• 44.
  Querying valid timetables SELECT EName, Edept FROM Emp WHERE ENo = 22217 AND EPeriod CONTAINS DATE '2015-01-23';
• 45.
  Querying valid timetables SELECT EName, Edept FROM Emp WHERE ENo = 31 AND EPeriod OVERLAPS PERIOD (DATE '2015-01-01', DATE '2015-01-31');
• 46.
  Transaction time ❖ Alsoknown as system time in SQL:2011 ❖ Modelled as two DATE or TIMESTAMP columns ❖ Management of the columns for the period is handled by the database for you
• 47.
  Transaction time ❖ Whendata is inserted: ❖ Start of transaction time is set to current time
• 48.
  Transaction time ❖ Whendata is updated: ❖ Transaction time end is set to current time on the existing row ❖ A new row is added with the updated date and a transaction time start of the current time
• 49.
  Transaction time ❖ Whendata is deleted: ❖ Transaction time end is set to current time in the existing row
• 50.
  Transaction time ❖ Becausethe system manages transaction time: ❖ Not possible to alter transaction time values in the past ❖ Not possible to add future dated transaction time values ❖ Referential constraints on historical data are never checked
• 51.
  Transaction time CREATE TABLEemp ( …, Sys_start TIMESTAMP(12) GENERATED ALWAYS AS ROW START, Sys_end TIMESTAMP(12) GENERATED ALWAYS AS ROW END, PERIOD FOR SYSTEM_TIME (Sys_start, Sys_end) ) WITH SYSTEM VERSIONING;
• 52.
  Querying transaction timetables ❖ New predicates to be used with transaction time: ❖ FOR SYSTEM_TIME AS OF ❖ FOR SYSTEM_TIME FROM ❖ FOR SYSTEM_TIME BETWEEN ❖ If none of the above supplied the database should only return rows for the current system time
• 53.
  Querying transaction timetables SELECT ENo, EName FROM emp WHERE Eno = 22;
• 54.
  Querying transaction timetables SELECT ENo, EName FROM emp WHERE ENo = 22 FOR SYSTEM_TIME AS OF TIMESTAMP '2015-01-28 12:45:00';
• 55.
  Querying transaction timetables SELECT ENo, EName FROM emp WHERE ENo = 22 AND EPeriod CONTAINS DATE '2014-08-27' FOR SYSTEM_TIME AS OF TIMESTAMP '2015-01-28 12:45:00';
• 56.
  Grey areas/not implementedyet ❖ Evolving schema over time ❖ Support for period joins ❖ Support for period aggregates or period grouped queries ❖ Support for period normalization ❖ Support for multiple valid time periods per table
• 57.
  Which vendors supportSQL: 2011?
• 58.
  Current support ❖ Oracle12c ❖ SQL:2011 compliant but not even nearly complete ❖ PostgreSQL ❖ 9.1 and earlier: temporal contributed package ❖ 9.2 native ranged data types ❖ IBM DB2 through ‘time travel query’ feature ❖ Teradata 13.10 and 14 ❖ Handful of others implemented as extensions
• 59.
  How do Iadd this stuff to my current schema?
• 60.
  Implementing valid time ❖Add a pair of date time columns to your table for the valid time period. ❖ Can make these part of your primary key
• 61.
  Implementing valid time ❖Things to consider: ❖ Have to check for end time > start time ❖ Have to check for overlaps in valid time periods ❖ Temporal foreign keys have to be implemented yourself ❖ Queries become potentially more complex
• 62.
  Implementing transaction time ❖Add a column recording transaction time start to your table ❖ For each table create a backup table mirroring the columns in the main table, adding a transaction time end column too ❖ Create a trigger that fires on each update or delete to copy old values from the main table to the backup table ❖ Should add transaction time end to the backup table ❖ Should also update the transaction time start to now in the main table if the operation is an update
• 63.
  Implementing transaction time ❖Things to consider: ❖ Extra complexity ❖ How long should backup data be kept for? ❖ Do you optimize for fast reads or writes? ❖ Should truncating the main table delete the data from the backup?
• 64.
  More information ❖ Wikipediaarticle on Temporal Databases ❖ Temporal features in SQL:2011 (PDF) ❖ Time and Relational Theory
• 65.
  Thanks for listening! ❖Any questions? ❖ I’d love some feedback ❖ https://joind.in/talk/view/13294 ❖ Contact me: ❖ @JCook21 ❖ [email protected]