HBase Data Types

HBase Data Types Nick Dimiduk, Hortonworks @xefyr n10k.com

Agenda

•  Motivations •  Progress thus far •  Future work •  Examples •  More Examples

2014-‐11-‐18 2 Licensed under a Crea3ve Commons A8ribu3on-‐ShareAlike 3.0 Unported License.

Why introduce types?

•  Δ(SQL, byte[]): (╯°□°）╯︵ ┻━┻ •  Rule of least surprise •  Interoperability across tools •  Distill best practices


Considerations

•  Opt-in for current users •  Easy transition for existing applications •  Client-side only mostly –  Filters, Split policies, Coprocessors, Block encoding

•  Avoid POJO constraints –  No required base-class/interface –  No magic (avoid ASM, ORM)

•  Non-Java clients •  HBASE-8089


Inspiration

•  Orderly •  PostgreSQL / PostGIS

•  HBASE-7221 •  HBASE-7692


Features: Encoding

•  Order preservation •  Override direction (ASC/DSC) •  Fixed, variable-width •  Null-able •  Self-identifying •  Efficient


Features: API

•  Complex type encoding – Compound rowkey pattern – Order preservation – Nullable fields

•  Runtime metadata •  User-extensible


Implementation$HBASE-8089

Implementation: Encoding


o.a.h.h.util.Bytes

•  numeric •  boolean •  int16, int32, int64 •  float32, float64 •  variable-length text

o.a.h.h.util.OrderedBytes

•  null •  numeric, +/-Inf, NaN •  int8, int16, int32, int64 •  float32, float64 •  variable-length text •  variable-length blob

Implementation: API


interface DataType<T>

•  decode() •  encode() •  encodedClass() •  encodedLength() •  getOrder() •  isNullable() •  isOrderPreserving() •  isSkippable() •  skip()

implements DataType

•  OrderedXXX •  RawXXX •  Struct –  StructBuilder –  StructIterator –  TerminatedWrapper –  FixedLengthWrapper

•  Union{2,3,4}

Up Next

•  “Default” types •  More complex types

–  Arrays/Lists –  Maps/Dicts

•  Tool integration –  Apache Phoenix –  Cloudera Kite

•  Performance audit, HBASE-8694 •  Improved metadata,

HBASE-8863 –  isCastableTo –  isCoercableTo –  isComparableTo

•  TypedTable, HBASE-7941 •  Beyond Java, HBASE-10091

–  REST –  Thrift –  Shell

•  ImportTsv, HBASE-8593 •  User documentation •  Coprocessors? •  Filters? •  CAS? •  DataBlockEncoders?


Examples

A case for TypedTable

Put p = new Put(Bytes.toBytes(u.user));

p.add(INFO_FAM, USER_COL, Bytes.toBytes(u.user));

p.add(INFO_FAM, NAME_COL, Bytes.toBytes(u.name));

p.add(INFO_FAM, EMAIL_COL, Bytes.toBytes(u.email));

p.add(INFO_FAM, PASS_COL, Bytes.toBytes(u.password));


A case for TypedTable!

static final RawString ENC_STR = new RawString();!

static final RawLong ENC_LONG = new RawLong();!

--!

!

SimplePositionedByteRange pbr =!

new SimplePositionedByteRange(100);!

ENC_STR.encode(pbr, u.user);!

Put p = new Put(Bytes.copy(pbr.getBytes(), pbr.getOffset(), pbr.getPosition()));!

p.add(INFO_FAM, USER_COL, Bytes.copy(pbr.getBytes(), ...);!

pbr.setPosition(0);!

ENC_STR.encode(pbr, u.name);!

p.add(INFO_FAM, NAME_COL, Bytes.copy(pbr.getBytes(), ...);!

...!


Structs: writing

!

!

!

Struct struct = new StructBuilder()!

.add(OrderedNumeric.ASCENDING)!

.add(OrderedString.ASCENDING)!

.toStruct();!

PositionedByteRange buf1 =!

new SimplePositionedByteRange(7);!

struct.encode(buf1,!

new Object[] { BigDecimal.ONE, "foo" });!

!


Structs: reading

!

!

!

!

buf1.setPosition(0);!

StructIterator it = longer.iterator(buf1);!

while (it.hasNext()) {!

System.out.print(it.next() + ", ");!

}!

!

> BigDecimal.ONE, foo!


Structs: schema migration

Struct addedFields = new StructBuilder()!





.toStruct();!

!

buf1.setPosition(0);!

StructIterator it = longer.iterator(buf1);!

while (it.hasNext()) {!

System.out.print(it.next() + ", ");!

}!

> BigDecimal.ONE, foo, null, null!

!2014-‐11-‐18 18 Licensed under a Crea3ve Commons A8ribu3on-‐ShareAlike 3.0 Unported License.

Protobuf (HBASE-11161)

!

class PBKeyValue extends PBType<CellProtos.KeyValue> {!

!

@Override!

public int encode(PositionedByteRange dst, KeyValue val) {!

CodedOutputStream os = outputStreamFromByteRange(dst);!

int before = os.spaceLeft(), after, written;!

val.writeTo(os);!

after = os.spaceLeft();!

written = before - after;!

dst.setPosition(dst.getPosition() + written);!

return written;!

}!


More Examples$https://gist.github.com/ndimiduk/bcf33f09cc7e4408f684

Thanks!

M A N N I N G

Nick Dimiduk Amandeep Khurana

FOREWORD BY Michael Stack

hbaseinaction.com

Nick Dimiduk github.com/ndimiduk

@xefyr

n10k.com

http://s.apache.org/bGN


HBase Data Types

Technology

Transcript of HBase Data Types