Silk: A Framework for Building Data Processing Workflows

Silk is an open-source workflow description and execution framework, written in Scala. With Silk you can create a workflow consisting of SQL, Unix commands, Scala programs.


def appleStock = nasdaq.filter(_.symbol is "APPL")
           .select(_.time, _.close)

// show the latest 10 stock price information


  • DSL
    • for building SQL statements
    • Define dependencies
    • Schedule definition
    • Stream processing operators
  • Weaver (Workflow executor)
    • Local executor
    • Remote executor


Object-oriented workflow programming

A workflow in Silk is a class (or trait in Scala). As in object-oriented programming style, you can encapsulate complex workflows within a class, override tasks in the workflow and reuse existing workflows.


  • Build SQL + local analysis workflows
  • Submit queries to Presto / Treasure Data
  • Run scheduled queries
  • Retry upon failures
  • Cache intermediate results
  • Resume workflow
  • Partial workflow executions
  • Sampling display
    • Interactive mode
  • Split a large query into small ones
    • Differential computation for time-series data
  • Windowing for stream queries

  • Input Source: fluentd/embulk
  • Output Source:

  • Workflow Executor
    • Local-only mode
    • Register SQL part to Treasure Data
    • UNIX command executor