Writing DICE Tests

Anatomy of a DICE Project

The file structure of a basic DICE project likes:

project_root
|-- oracles
|   `-- pyramid.yaml
`-- utils
    `-- item.py
  • An item.py is a python script defines how a single test item is run and pass the result to DICE for analysis.
  • The oracles directory contains one or more YAML files defines the expected results for different conditions.
  • The optional utils directory contains python helper modules to assist specific tests.

Writing Test Runner

item.py contains a class Item inherits from dice.item class from DICE`s core API:

import os

from dice.core import item
from dice import utils


class Item(item.ItemBase):
    def run(self):
        cmdline = os.path.join(self.provider.path, 'pyramid')
        cmdline += ' %s' % utils.escape(str(self.get('option')))
        self.res = utils.run(cmdline)

Writing Oracle

An example oracle YAML file likes:

- name: option
  oracle: |
      if option is Integer:
          if option < 0:
              return FAIL('Min input is 0')
          elif option > 9223372036854775808:
              return FAIL('Number overflow')
          elif option > 1000:
              return FAIL('Max input is 1000')
          else:
              return SUCCESS()
      else:
          return FAIL('Invalid number')

Every oracle YAML file contains a list of oracle objects. For each oracle, there is some predefined properties.

  • name is the identifier of a specific oracle. It is recommended in CamelCase style to be differentiated from other variables.
  • target is where this oracle is applied to for the test item.
  • tree is a python style code snippet shows the expected result for a given conditions.