Big Data

• 3V: high Volume, Velocity, Variety
• in real world: gov, industry
• modren: VLDB(Very Large DataBase)
  • parallel processing 중요해짐
  • ex. IPA: 10x faster, 6x less memory, 20k times accurate, easily parallelized
• Storage
  • Centralized Storage
    • expensive
    • fast transfer
  • Distributed Storage
    • inexpensive
    • slow
    • → new programming to Minimize data transfer
      • MapReduce: move operations, instead of data
        • run operation on data nodes
• AI vs Big Data
  • AI
    • image(CV)
    • language(NLP)
  • Big Data
    • Table, Graph, Matrix, Tensor, Sequence, …

Data Type

• Table
  • records(row, 행), attributes(column, 열)
    • ex. relational database
• vector, matrix, tensor
  • multi-dim
• graph, network
  • nodes and edges
    • ex. web structure
• sequence, ordered data
  • ex. DNA
• time-series
  • ex. sensor, stream data
• Image and multimedia
  • ex. pictures, videos
• Spacial data
  • ex. map, spatiotemporal(space and time) data

Data Model = Structure+Constraint+Operation

• Structures
  • rows, columns
  • nodes, edges
  • key-value pairs
  • sequence of bytes
• Constraints
  • all rows have same number of columns
  • all values in col have same type
  • a child cannot have two parents(tree)
• Operations
  • find the value x
  • …