• High-Level

    A high level description of this document, which is meant to be used as a reference for the development of madz. This will briefly cover many of the main points of the entire document.

  • Concepts

    Important concepts required for understanding the vision and design of madz. These will be presented in such a way as to only rely on previous concepts. But will assume the ability to look up well known concepts, specifically including:

    • Ontologies
    • Contexts
    • Entity/Component
  • Vision

    Some examples and descriptions of what madz should provide. Vision is split into different categories of what madz will provide.

    The core point of madz is to manage software projects. This ranges from checking out the source code, exploring (aka ‘reading’) the source code, organizing the source code, adding to the source code, editing the source code, committing the source code, building the source code, testing the source code, deploying the source code, debugging the source code, reproducing errors with the source code, documenting the source code, importing external source code, and exporting existing source code and any other source code manipulation.

    It also provides features for managing libraries, SDKs and APIs, and custom tool chains used by the developer.

  • Design

    A highlevel discussion of feature design, at multiple levels of madz. Design is split into abstraction layers, each of which represents a set of abstractions used by components and systems of madz at that layer, as well as the implementation of features at that layer.

    Some of the abstraction layers described here will in reality be separate libraries originally designed for madz.

  • What is Massive Dangerzone

    Abbreviated madz, massive dangerzone is a system for managing data, and data about that data.

    More specifically it is an attempt to fix many common problems in software engineering by re-imaging how we manage data. It is a framework for building software systems. It’s initial primary usage is as a tool chain for managing complex code bases.

  • Purpose

    The guiding purpose of madz is, simply, to: Fix Software Engineering. There are a number of core principles which unpack from that. In no particular order.

  • Insights

    There are a couple of core realizations which madz takes for granted. Some of these are architecture related, others are high level concepts, and some aren’t easily categorized.

  • Technologies

    These are the core technologies required for madz. They will start as python libraries within madz, and eventually be separated into madz module libraries.

  • Module

    The primary purpose of madz is to manage modules. That is data, and data about that data. Modules can have many different properties, some of which will be covered below as concepts. Different sets of properties are used by madz to provides different features for modules.

    Modules can go by many names depending on the properties it has, note the names are generally not exclusive (although some are subsets of others), hence modules are an ontology (as opposed to a taxonomy).

  • World/Noun/Verb

    Markers within the module system which keep track of objects and actions, and how they relate to each other.

    Specifically the entity system for the modules keeps track of the interesting objects related to each module, and other objects register themselves as interesting in other ways, each registration contains a description object which allows the objects to be organized, typed, and compared.

    Verbs are also added to the system, whether as compound actions which invoke other verbs, or as singular verbs. These verbs can provide different capabilities and information. For example descriptions of the types of objects they require, alternate functions which can pretend to do the action if given mocked up object types, etc. Others may out right describe the action they perform.

    Verbs always take a world, every noun is part of a world. Many nouns provide functionality for creating mocked versions in a ‘fake’ world. World use contexts.

    This system is used by the AI to perform actions across modules. And by the user interface for generating interactive consoles and completing input.

  • Description

    Module

    Every module has a description. This description includes information about the purpose of the module, how to use it, and how it should affect other parts of the system. The description is what drives the majority of the other features of the module system.

    Each description has an arbitrary number of sub parts, depending on the features available for parsing it. Each description component has a list of dependencies from which it comes from. Descriptions are acquired in various stages.

    It’s important to note that types of information generated, and the formats containing information are disjoint. One file could contain identity, forge, and code descriptions. Description information can also be inherited from libraries (calculated at reduce time).

  • Identity

    Module

    Perhaps the most important part of a module is it’s identity. However identities exist for many other things. For example, both forges (and their components) and code symbols have identities.

    An identity provides a couple of key features:

    • Searching: Identities provide a way to search for an instance of something, based on any of it’s properties.
    • Namespacing: Identities provide organizational information which lets them be organized into ontologies of instances.
    • Versioning: Identities provides the ability to differentiate between different versions of the same thing. But also for choosing and requesting a specific version of something.
    • Security: Identities provide security that the plugin you are using is safe to use, based off of the trusted authorities for each plugin group.
    • Uniqueness: Identities are unique, and provide ways of matching range identities to unique instances.

    When used for code symbols, for example, the identity system is used to find the symbol in question, and other partial matches.

    When used for forges, for example, the identity system is used for searching, versioning, and compatibility.

    Finally, when used for module, for example, the identity system provides searching and matching.

  • Projection

    A projection is an operation performed on information (like a description or source code) which produces another view, or format, of the source. Typically projections are one way, although they don’t need to be. For our purposes projections means an operation which takes information in an in memory machine format (for example, in a parse tree, or the generated semantic tree) and produces another format meant for storage or reading.

  • Artifact

    An artifact is the projection of some of the description and source code of a plugin, from which the description and source is likely not recoverable, and more specifically, which is designed to be executable. That is an executable projection.

  • Forge

    Forges are a collection of code which is responsible for generating artifacts for code modules. A forge may be responsible for multiple code modules, and multiple types of artifacts.

    Forges contain a lot of parts, specifically:

    • Wrapper Generator Fragments.
    • Compiler Configurators.
    • Linker Configurators.
    • Compilation Managers.
    • Protocol Endpoint Configurators.
    • Library Configurators.
    • Runtime Configurators.
    • Interpreter Configurators.

    Generally the design of the forge is that some parts provide new Actions for the plugins whose forge identities match a forge range. These actions are often limited to the target being attempted.

  • Reports

    Reports represent another entity component system. Specifically a way of describing the result of some action. Reports are split into trees of results of the action. And provide human readable data. And potentially internal madz references to other reports or available features.

    They are meant to provide a dynamic way for people to inspect the results of madz actions, and view the current state.

    • Top-Level Organization

      Madz is not a program, and it is not an application, at least not in the traditional sense. Madz is a library for managing data, with many high level objects cooperating to deliver powerful features.

      Generally madz will have a toplevel set of bindings which represent the entire context of the current madz execution. These are global variables (actually thread local to allow for concurrent execution) which provide the top level objects of madz which orchestrate it’s features.

      These toplevel bindings may be manipulated to provide different environments as required.

      The list of top level objects:

      • Module Ecs (Bootstrap Ecs)
      • File Ecs
      • Report Ecs
      • Taskmaster
      • Forge Architect
      • Planner
    • Description Loading

      Description loading is a lazy, resume-able, multistage process to find descriptions of modules.

      • All registered description acquirers are executed across the format. These are arranged in a dependency tree, and acquire all of the possible information from a module. Often via inspection, unpacking, etc. (For example, file based acquirers will list the file listing description as a dependency). Once an acquirer has determined a module isn’t the primary one in question, it will pause until the module’s description is required (this will allow lazy networked module acquirers).
      • All registered description expanders are executed across the ‘base unexpanded description’ these expanders are meant to expand and reformat the existing information into other usable formats. In many cases, these formats will be lazy (being added, but only actually expanded if asked for). Also arranged in a dependency tree.
      • Once all modules related to the primary one have a ‘base expanded description’, all registered description reducers synthesize global objects containing indexes, and combined information about descriptions, additionally base descriptions may be updated to contain references into the synthesized objects (this is domain dependent). It’s at this point that inconsistencies are detected, and hopefully, resolved. Also executed in a dependency tree.
    • Modularity

      Software engineers love modularity, but we rarely see it in our support tools. How modular is your compiler? Your programming languages? Your build system? These are the tools used every day to build software, but to add, remove, or change features in them is ridiculously difficult.

      Hence a core principle is to make every aspect modular. Modular abstraction capabilities, modular implementation features, modular optimization techniques, even a modular framework of modularity.

    • Powerful Abstraction

      The core process of software engineering is the generation and implementation of abstractions. However there are few (if any) ways to represent abstraction across multiple domains, capturing multiple, potentially unlimited, facets, in arbitrary ways.

      Let alone ways to translate these abstractions to different equivalent domains, and projecting them onto arbitrary limited information domains. We use design patterns to reduce the mental overhead, but they don’t fix the root problem and cause lost information.

      Hence a core principle is to enable powerful abstraction description and manipulation. Design patterns should be a first class objects.

      This is in opposition to monolithic, opinionated, incremental, or platform specific abstraction systems.

    • Variant Implementation

      The other side of the core process of software engineering is implementation. The implementation of abstraction(s) is a process of choosing between a myriad of competing concerns. Two implementations of the same concept are unlikely to be exactly the same. Yet there are few effective ways for managing variant implementations of the same concept at all but the most trivial levels.

      Hence a core principle is to enable multiple implementations for any abstraction. Reimplementation should be done side by side, and should be separable from the project with the original implementation.

      This is in opposition to concrete implementation designs.

    • Arbitrary Platform

      How many projects use more than a couple programming languages? Or even more than one? Why is it so hard to setup a new programming language in your current project. If part A of the project requires features that language X excels at, and part B requires features that language Y excels at. Why choose language Z which is simply good for both A and B.

      This applies to drivers (and the libraries they provide; e.g. DirectX vs OpenGl), operating systems, run-times, virtual machines, and communication protocols.

      Hence a core principle is to allow any platform to be used for any implementation. Though this may limit which platforms the final result is usable on, and reduce performance.

      This is in opposition to platform limited tool chains (they all are, madz will also provide “madz free” platforms, allowing it to be used for arbitrary projects).

    • Holistic Optimization

      Software projects should be considered in their entirety when optimizing. Optimizations should be possible across multiple layers of abstractions, considering all possible implementations, and manipulations based on their platforms. Engineers should worry about writing locally optimal code, and directing high level machinery to glue it together optimally. Where optimal is defined as the concerns of the engineers for each portion of the software.

      Hence a core principle is to optimize across all available data.

      This is in opposition to limited optimizations done by many toolchains.

    • Universal Identity

      The identity of something should never be in flux. It should be known exactly what you are asking for. Even if what you are asking for is a partial identity. Too many toolchains have ill-defined ways of managing the identity of code. Duplicate version numbers, code forks, abstraction changes.

      Hence a core principle is to provide capabilities for useful universal identity.

    • Ontologies

      We use ontologies to represent a number of objects within our system because they provide a way for something to be of many different types, which allows us to only focus on the relevant classification information, and allow other information to exist separately from that which our current section of the system can understand.

    • Metadata

      One of our core insights is that data should have useable metadata, that is data about the data. Our core goal is to build a system which can, when given some data, understand it by using the data about the data.

      This comes back to bootstrapping, the pre-bootstrap system can understand specific types of data, which it can use to understand other data, and so on, arbitrarily, until the goal is reached.

    • AI

      By using AI techniques like Planning we can find the solutions to the problems facing the system. For example, we can use a knowledge base about our current information and a planner to find the correct piece of data or action to take.

    • Parse-Represent-Project Cycle

      Another core insight is having the ability to parse some data, learn about the representation of that data, and project it back out. Performing this cycle is a common operation within madz.

    • EntityComponentSystem

      (Massive-Dangerzone.Runtime.System.Entity-Component)

      A library for making entity component systems. Supports various patterns for manipulating entity component systems, including madz wrapper syntax extensions. Also provides interfaces for adding optimizations to entity system construction.

    • DependencyTree

      (Massive-Dangerzone.Utility.Dependency-Tree)

      A library for making entity component systems. Supports various patterns for manipulating entity component systems, including madz wrapper syntax extensions. Also provides interfaces for adding optimizations to entity system construction.

    • SemanticNode

      (Syndicate.Node)
      (Massive-Dangerzone.Description.Node)

      Uses EntityComponentSystem to provide a semantic node library. Primarily translations between different semantic domains, and support for tree/graph traversals of semantic nodes, among other utilities for support of end goal features like parsing and generation. Also includes serialization/serialization.

    • SemanticParsers

      (Syndicate.Parser)
      (Massive-Dangerzone.Description.Parser)

      A library for making parsers using semantic node. Will support the full range of context free grammars via Earley and SGLR based parsers. will include generalized techniques for optimizing context free rules (separate from string based parsing).

    • SemanticGenerators

      (Syndicate.Generator)
      (Massive-Dangerzone.Description.Generator)

      A library for generating well formatted text (and other outputs) using semantic nodes. Based broadly on the idea of hierarchical planning rules (but Context Free Limited), some of which can generate more nodes, others of which can actually generate strings.

    • SemanticLanguages

      (Syndicate.Language)
      (Massive-Dangerzone.Description.Language)

      A library for generating languages, i.e. both the parsers and generators and semantic node domain for a language. As a high level library, may also eventually include additional sub libraries, like configs, alternate (non-graphical) outputs, etc.

    • PlanningLogic

      (Massive-Dangerzone.Planning.Logic)

      A library of basic logic routines for use with planning. Namely declarative language like solutions. Including a description of solvers.

    • PlanningGeneral

      (Massive-Dangerzone.Planning.General)

      A library providing ways to specify general planning problems (i.e. actions, objects, predicates, heuristics). And modular solvers for them.

      Eventually including additions (potentially as separate libraries) such as:

      • HTN Methods as actions.
      • Ordered Actions (and methods).
      • Heuristic Ordering.
      • Multiple Solvers.
      • Live Solving.
    • Identity

      (Massive-Dangerzone.Runtime.System.Identity)

      A library providing features for providing identity of objects. Supports various patterns for manipulating identity systems, including madz wrapper syntax extensions. Can be mixed with entity components.

      Focuses on providing a way of uniquely identifying objects. Supporting techniques involving cryptography, unique identifiers, and human readable info. As well as generated human information.

    • Code Module

      The most common type of properties held by a module (‘module’ is often assumed to refer to code module, unless another name is used). Contains source code in a language suitable for compiling and integration with a binary.

    • Asset Module

      A module containing assets which aren’t code (or more specifically, not meant for inclusion with the binary). Asset modules often have their assets collected by another process and packed up separately from the binary.

    • Plugin [Module]

      A code module meant to be dynamically added at runtime, often dynamically updated as well. Note that a plugin can still be added at build time like a normal Code Module, but this may prevent some of it’s plugin features (like reloading) from being used correctly.

    • Library [Module]

      A module which provides organizational information, documentation, and other project level information for a collection of other modules.

    • Executable [Module]

      A module which describes entry point(s) for the project.

    • Macro [Module]

      A code module which provides a variety of code modules via it’s code. Sort of like a lisp macro but more expansive. Generally used for things like templates.

    • Adaptor [Module]

      A code module which adapts one module to provide another. Generally used to adapt between different variations or versions of existing modules.

    • Bridge [Module]

      A code module which adapts some other codebase to fit within a module.

    • Extension [Module]

      A code module which adds features via indirect inclusion (i.e. inversion of control, dependency injection, reflection).

    • Concept [Module]

      A module which contains only a description. (i.e. not an asset or a code module).

    • Interface [Module]

      A concept module specifically for the description of plugin modules.

    • Remote Plugin [Module]

      A plugin module which is dynamically loaded over a network, interprocess, or similar connection (i.e. the plugin is actually running on another machine).

    • Remote Bridge [Module]

      A bridge module which wraps a live service (like an API).

    • Identity Description

      The description of the identity of the plugin. Namely for versioning, namespacing, organizing, and searching. But probably most importantly for uniqueness and security. The identity description includes signature descriptions, and all of the identity domain descriptions can be used for uniqueness. Some example components within the identity description domain:

      • Custom Name(s)
      • Author(s)
      • (Cryptographic) Signatures.
      • Guids.
      • Organizational Locations (AKA, Namespaces).
      • Hash.
      • Previous Version:
        • Signature.
        • Hash.
        • Guid.
    • Dependency Description

      A description of the other modules this one relies on. Different modules are relied on in different ways, and those relationships may describe how to build other modules based off of the relationship.

    • Code Description

      This is a description of code, often including the code within this module, but could describe code in other modules. Often relies on identity to provide unique locations of some types within it.

      Namely this module is responsible for describing the way code modules interact, for example:

      • Types:
        • Abstract.
        • Concrete, in this module.
        • Object System Types.
        • Generic/Parametric Types.
        • Other System Types.
        • Type Relationships.
      • Variables, in this module.
      • Operators, across multiple modules.
    • Forge Description

      This is a description of how to build, wrap, load, and execute a code module. Relies on the code description and dependency description to do some operations (but not others). Namely contains:

      • Inter-module Protocol(s) used for which plugin dependencies.
      • Libraries(s) used.
      • Language(s) used. Which computes/contains:
        • Compiler(s) used, and it’s targets.
        • Linker(s) used, and it’s targets.
        • Interpreter(s) used, and it’s targets.
        • Runtime(s) used.
        • Wrapper features/providers used (also depends on the inter-module protocols and libraries).
    • Identity

      A singular identity is a collection of information describing a unique entity. Some of this information may be indexed, or otherwise used to provide features, but an identity is singular.

    • Identity Range

      An identity range describes a potential range of identities, and is attempting to find a single one. Often there are resolution methods attached to the range which can choose the best choice among a list of plugins.

    • Artifact Target

      Every artifact has a target, which is an identity range of what it needs to be useable on. The matching identity is the target actually being ran on. The target includes information like:

      • Dll Identity Ranges. Including version, name, security information. Often translated into an OS specific format as part of the artifact.
      • The hardware required, for example SSE extensions, and bit width.
    • Action Reports

      Nearly every piece of code which runs in madz is tied to an action, some driving force which caused it to be executed. Reports are hence organized primarily by the action that they were responsible for. This is the entity used in reports, the action that caused the code to be ran.

    • State Reports

      State Reports express the current state of the madz system, including the latest action reports in each area. For example every time a plugin is built, it’s latest action report is used as the current state of that plugin.

    • Object Reports

      A subset of state reports, these are collections of the latest action reports for a specific object.

    • Dynamic Reports

      Reports have the capability to be dynamic, to be updated when information changes, to provide the most up to date state of the system. And also to provide links to objects, and their actions.

    • Report Logs

      Perhaps the most commonly used part of reports are the logs they contain, which contain text, and objects, about what happened. Report logs contain tags, and other features, for organizing them.

    • Entity Component Systems

      Entity component systems represent the live data stores for many of the more dynamic entity types, or more precisely, for ontological (as opposed to taxonomical) based object systems. Primarily modules (and the bootstrapped modules), although reports and files are also setup as entity component systems.

      These systems are designed to provide objects which are ontological in nature, and provide support structures for those objects to act like any object required.

    • Description Detectors

      Detect new module descriptions.

      {"cards":[{"_id":"4130fbe49d3b2c4d8b00001e","treeId":"4130f67a9d3b2c4d8b00001b","seq":1,"position":1,"parentId":null,"content":"# High-Level\n\nA high level description of this document, which is meant to be used as a reference for the development of madz. This will briefly cover many of the main points of the entire document."},{"_id":"413105a09d3b2c4d8b000021","treeId":"4130f67a9d3b2c4d8b00001b","seq":318500,"position":0.5,"parentId":"4130fbe49d3b2c4d8b00001e","content":"## What is Massive Dangerzone\n\nAbbreviated madz, massive dangerzone is a system for managing data, and data about that data.\n\nMore specifically it is an attempt to fix many common problems in software engineering by re-imaging how we manage data. It is a framework for building software systems. It's initial primary usage is as a tool chain for managing complex code bases."},{"_id":"4130fd599d3b2c4d8b00001f","treeId":"4130f67a9d3b2c4d8b00001b","seq":318504,"position":1,"parentId":"4130fbe49d3b2c4d8b00001e","content":"## Purpose\n\nThe guiding purpose of madz is, simply, to: **Fix Software Engineering**. There are a number of core principles which unpack from that. In no particular order."},{"_id":"41316a879d3b2c4d8b000023","treeId":"4130f67a9d3b2c4d8b00001b","seq":318506,"position":0.5,"parentId":"4130fd599d3b2c4d8b00001f","content":"### Modularity\n\nSoftware engineers love modularity, but we rarely see it in our support tools. How modular is your compiler? Your programming languages? Your build system? These are the tools used every day to build software, but to add, remove, or change features in them is ridiculously difficult.\n\nHence a core principle is to **make every aspect modular**. Modular abstraction capabilities, modular implementation features, modular optimization techniques, even a modular framework of modularity."},{"_id":"4131051e9d3b2c4d8b000020","treeId":"4130f67a9d3b2c4d8b00001b","seq":1,"position":1,"parentId":"4130fd599d3b2c4d8b00001f","content":"### Powerful Abstraction\n\nThe core process of software engineering is the generation and implementation of abstractions. However there are few (if any) ways to represent abstraction across multiple domains, capturing multiple, potentially unlimited, facets, in arbitrary ways.\n\nLet alone ways to translate these abstractions to different equivalent domains, and projecting them onto arbitrary limited information domains. We use design patterns to reduce the mental overhead, but they don't fix the root problem and cause lost information.\n\nHence a core principle is to **enable powerful abstraction description and manipulation**. Design patterns should be a first class objects.\n\nThis is in opposition to monolithic, opinionated, incremental, or platform specific abstraction systems."},{"_id":"413125c89d3b2c4d8b000022","treeId":"4130f67a9d3b2c4d8b00001b","seq":1,"position":2,"parentId":"4130fd599d3b2c4d8b00001f","content":"### Variant Implementation\n\nThe other side of the core process of software engineering is implementation. The implementation of abstraction(s) is a process of choosing between a myriad of competing concerns. Two implementations of the same concept are unlikely to be exactly the same. Yet there are few effective ways for managing variant implementations of the same concept at all but the most trivial levels.\n\nHence a core principle is to **enable multiple implementations for any abstraction**. Reimplementation should be done side by side, and should be separable from the project with the original implementation.\n\nThis is in opposition to concrete implementation designs."},{"_id":"413191e19d3b2c4d8b000024","treeId":"4130f67a9d3b2c4d8b00001b","seq":1,"position":3,"parentId":"4130fd599d3b2c4d8b00001f","content":"### Arbitrary Platform\n\nHow many projects use more than a couple programming languages? Or even more than one? Why is it so hard to setup a new programming language in your current project. If part A of the project requires features that language X excels at, and part B requires features that language Y excels at. Why choose language Z which is simply good for both A and B.\n\nThis applies to drivers (and the libraries they provide; e.g. DirectX vs OpenGl), operating systems, run-times, virtual machines, and communication protocols.\n\nHence a core principle is to **allow any platform to be used for any implementation**. Though this may limit which platforms the final result is usable on, and reduce performance.\n\nThis is in opposition to platform limited tool chains (they all are, madz will also provide \"madz free\" platforms, allowing it to be used for arbitrary projects)."},{"_id":"4131a88b9d3b2c4d8b000025","treeId":"4130f67a9d3b2c4d8b00001b","seq":1,"position":4,"parentId":"4130fd599d3b2c4d8b00001f","content":"### Holistic Optimization\n\nSoftware projects should be considered in their entirety when optimizing. Optimizations should be possible across multiple layers of abstractions, considering all possible implementations, and manipulations based on their platforms. Engineers should worry about writing locally optimal code, and directing high level machinery to glue it together optimally. Where optimal is defined as the concerns of the engineers for each portion of the software.\n\nHence a core principle is to **optimize across all available data**.\n\nThis is in opposition to limited optimizations done by many toolchains."},{"_id":"4131c4229d3b2c4d8b000026","treeId":"4130f67a9d3b2c4d8b00001b","seq":1,"position":5,"parentId":"4130fd599d3b2c4d8b00001f","content":"### Universal Identity\n\nThe identity of something should never be in flux. It should be known exactly what you are asking for. Even if what you are asking for is a partial identity. Too many toolchains have ill-defined ways of managing the identity of code. Duplicate version numbers, code forks, abstraction changes.\n\nHence a core principle is to **provide capabilities for useful universal identity**."},{"_id":"413d0a15f8e2645dad000021","treeId":"4130f67a9d3b2c4d8b00001b","seq":1,"position":2,"parentId":"4130fbe49d3b2c4d8b00001e","content":"## Insights\n\nThere are a couple of core realizations which madz takes for granted. Some of these are architecture related, others are high level concepts, and some aren't easily categorized."},{"_id":"413d0fbcf8e2645dad000022","treeId":"4130f67a9d3b2c4d8b00001b","seq":318673,"position":1,"parentId":"413d0a15f8e2645dad000021","content":"### Ontologies\n\nWe use ontologies to represent a number of objects within our system because they provide a way for something to be of many different types, which allows us to only focus on the relevant classification information, and allow other information to exist separately from that which our current section of the system can understand."},{"_id":"461699aeb10ec6a401000091","treeId":"4130f67a9d3b2c4d8b00001b","seq":318675,"position":1.5,"parentId":"413d0a15f8e2645dad000021","content":"### Metadata\n\nOne of our core insights is that data should have useable metadata, that is data about the data. Our core goal is to build a system which can, when given some data, understand it by using the data about the data.\n\nThis comes back to bootstrapping, the pre-bootstrap system can understand specific types of data, which it can use to understand other data, and so on, arbitrarily, until the goal is reached."},{"_id":"413d29a4f8e2645dad000023","treeId":"4130f67a9d3b2c4d8b00001b","seq":318677,"position":2,"parentId":"413d0a15f8e2645dad000021","content":"### AI\n\nBy using AI techniques like Planning we can find the solutions to the problems facing the system. For example, we can use a knowledge base about our current information and a planner to find the correct piece of data or action to take."},{"_id":"413d36b6f8e2645dad000024","treeId":"4130f67a9d3b2c4d8b00001b","seq":318679,"position":3,"parentId":"413d0a15f8e2645dad000021","content":"### Parse-Represent-Project Cycle\n\nAnother core insight is having the ability to parse some data, learn about the representation of that data, and project it back out. Performing this cycle is a common operation within madz."},{"_id":"413d3e4bf8e2645dad000025","treeId":"4130f67a9d3b2c4d8b00001b","seq":1,"position":3,"parentId":"4130fbe49d3b2c4d8b00001e","content":"## Technologies\n\nThese are the core technologies required for madz. They will start as python libraries within madz, and eventually be separated into madz module libraries."},{"_id":"41667159dd5490308d000028","treeId":"4130f67a9d3b2c4d8b00001b","seq":1,"position":0.5,"parentId":"413d3e4bf8e2645dad000025","content":"### EntityComponentSystem\n##### (Massive-Dangerzone.Runtime.System.Entity-Component)\n\nA library for making entity component systems. Supports various patterns for manipulating entity component systems, including madz wrapper syntax extensions. Also provides interfaces for adding optimizations to entity system construction."},{"_id":"416777c8dd5490308d000045","treeId":"4130f67a9d3b2c4d8b00001b","seq":1,"position":0.75,"parentId":"413d3e4bf8e2645dad000025","content":"### DependencyTree\n##### (Massive-Dangerzone.Utility.Dependency-Tree)\n\nA library for making entity component systems. Supports various patterns for manipulating entity component systems, including madz wrapper syntax extensions. Also provides interfaces for adding optimizations to entity system construction."},{"_id":"413d3ec1f8e2645dad000026","treeId":"4130f67a9d3b2c4d8b00001b","seq":1,"position":1,"parentId":"413d3e4bf8e2645dad000025","content":"### SemanticNode\n##### (Syndicate.Node)\n##### (Massive-Dangerzone.Description.Node)\nUses EntityComponentSystem to provide a semantic node library. Primarily translations between different semantic domains, and support for tree/graph traversals of semantic nodes, among other utilities for support of end goal features like parsing and generation. Also includes serialization/serialization."},{"_id":"4166a546dd5490308d00002a","treeId":"4130f67a9d3b2c4d8b00001b","seq":1,"position":2,"parentId":"413d3e4bf8e2645dad000025","content":"### SemanticParsers\n##### (Syndicate.Parser)\n##### (Massive-Dangerzone.Description.Parser)\nA library for making parsers using semantic node. Will support the full range of context free grammars via Earley and SGLR based parsers. will include generalized techniques for optimizing context free rules (separate from string based parsing)."},{"_id":"4166ab43dd5490308d00002b","treeId":"4130f67a9d3b2c4d8b00001b","seq":1,"position":3,"parentId":"413d3e4bf8e2645dad000025","content":"### SemanticGenerators\n##### (Syndicate.Generator)\n##### (Massive-Dangerzone.Description.Generator)\n\nA library for generating well formatted text (and other outputs) using semantic nodes. Based broadly on the idea of hierarchical planning rules (but Context Free Limited), some of which can generate more nodes, others of which can actually generate strings."},{"_id":"4166b6bddd5490308d00002c","treeId":"4130f67a9d3b2c4d8b00001b","seq":1,"position":4,"parentId":"413d3e4bf8e2645dad000025","content":"### SemanticLanguages\n##### (Syndicate.Language)\n##### (Massive-Dangerzone.Description.Language)\n\nA library for generating languages, i.e. both the parsers and generators and semantic node domain for a language. As a high level library, may also eventually include additional sub libraries, like configs, alternate (non-graphical) outputs, etc."},{"_id":"4166b8afdd5490308d00002d","treeId":"4130f67a9d3b2c4d8b00001b","seq":1,"position":5,"parentId":"413d3e4bf8e2645dad000025","content":"### PlanningLogic\n##### (Massive-Dangerzone.Planning.Logic)\n\nA library of basic logic routines for use with planning. Namely declarative language like solutions. Including a description of solvers."},{"_id":"4166c0d4dd5490308d00002e","treeId":"4130f67a9d3b2c4d8b00001b","seq":1,"position":6,"parentId":"413d3e4bf8e2645dad000025","content":"### PlanningGeneral\n##### (Massive-Dangerzone.Planning.General)\nA library providing ways to specify general planning problems (i.e. actions, objects, predicates, heuristics). And modular solvers for them.\n\nEventually including additions (potentially as separate libraries) such as:\n* HTN Methods as actions.\n* Ordered Actions (and methods).\n* Heuristic Ordering.\n* Multiple Solvers.\n* Live Solving."},{"_id":"4166cc25dd5490308d000031","treeId":"4130f67a9d3b2c4d8b00001b","seq":1,"position":7,"parentId":"413d3e4bf8e2645dad000025","content":"### Identity\n##### (Massive-Dangerzone.Runtime.System.Identity)\n\nA library providing features for providing identity of objects. Supports various patterns for manipulating identity systems, including madz wrapper syntax extensions. Can be mixed with entity components.\n\nFocuses on providing a way of uniquely identifying objects. Supporting techniques involving cryptography, unique identifiers, and human readable info. As well as generated human information."},{"_id":"4166d509dd5490308d000032","treeId":"4130f67a9d3b2c4d8b00001b","seq":1,"position":1.5,"parentId":null,"content":"# Concepts\n\nImportant concepts required for understanding the vision and design of madz. These will be presented in such a way as to only rely on previous concepts. But will assume the ability to look up well known concepts, specifically including:\n\n* Ontologies\n* Contexts\n* Entity/Component"},{"_id":"4166d6f5dd5490308d000033","treeId":"4130f67a9d3b2c4d8b00001b","seq":318520,"position":1,"parentId":"4166d509dd5490308d000032","content":"## Module\n\nThe primary purpose of madz is to manage modules. That is data, and data about that data. Modules can have many different properties, some of which will be covered below as concepts. Different sets of properties are used by madz to provides different features for modules.\n\nModules can go by many names depending on the properties it has, note the names are generally not exclusive (although some are subsets of others), hence modules are an ontology (as opposed to a taxonomy)."},{"_id":"41673a3ddd5490308d000034","treeId":"4130f67a9d3b2c4d8b00001b","seq":1,"position":1,"parentId":"4166d6f5dd5490308d000033","content":"### Code Module\nThe most common type of properties held by a module ('module' is often assumed to refer to code module, unless another name is used). Contains source code in a language suitable for compiling and integration with a binary."},{"_id":"41673a8bdd5490308d000035","treeId":"4130f67a9d3b2c4d8b00001b","seq":1,"position":2,"parentId":"4166d6f5dd5490308d000033","content":"### Asset Module\nA module containing assets which aren't code (or more specifically, not meant for inclusion with the binary). Asset modules often have their assets collected by another process and packed up separately from the binary."},{"_id":"41673aecdd5490308d000036","treeId":"4130f67a9d3b2c4d8b00001b","seq":1,"position":3,"parentId":"4166d6f5dd5490308d000033","content":"### Plugin [Module]\nA code module meant to be dynamically added at runtime, often dynamically updated as well. Note that a plugin can still be added at build time like a normal Code Module, but this may prevent some of it's plugin features (like reloading) from being used correctly.\n"},{"_id":"41673b42dd5490308d000037","treeId":"4130f67a9d3b2c4d8b00001b","seq":1,"position":4,"parentId":"4166d6f5dd5490308d000033","content":"### Library [Module]\nA module which provides organizational information, documentation, and other project level information for a collection of other modules."},{"_id":"416754b0dd5490308d000041","treeId":"4130f67a9d3b2c4d8b00001b","seq":1,"position":4.5,"parentId":"4166d6f5dd5490308d000033","content":"### Executable [Module]\n\nA module which describes entry point(s) for the project."},{"_id":"41673c6cdd5490308d000038","treeId":"4130f67a9d3b2c4d8b00001b","seq":1,"position":5,"parentId":"4166d6f5dd5490308d000033","content":"### Macro [Module]\n\nA code module which provides a variety of code modules via it's code. Sort of like a lisp macro but more expansive. Generally used for things like templates."},{"_id":"41673ea7dd5490308d000039","treeId":"4130f67a9d3b2c4d8b00001b","seq":1,"position":6,"parentId":"4166d6f5dd5490308d000033","content":"### Adaptor [Module]\n\nA code module which adapts one module to provide another. Generally used to adapt between different variations or versions of existing modules."},{"_id":"4167443add5490308d00003a","treeId":"4130f67a9d3b2c4d8b00001b","seq":1,"position":7,"parentId":"4166d6f5dd5490308d000033","content":"### Bridge [Module]\n\nA code module which adapts some other codebase to fit within a module."},{"_id":"41674636dd5490308d00003c","treeId":"4130f67a9d3b2c4d8b00001b","seq":1,"position":8,"parentId":"4166d6f5dd5490308d000033","content":"### Extension [Module]\n\nA code module which adds features via indirect inclusion (i.e. inversion of control, dependency injection, reflection)."},{"_id":"4167476cdd5490308d00003d","treeId":"4130f67a9d3b2c4d8b00001b","seq":1,"position":9,"parentId":"4166d6f5dd5490308d000033","content":"### Concept [Module]\n\nA module which contains only a description. (i.e. not an asset or a code module)."},{"_id":"41674bcfdd5490308d00003e","treeId":"4130f67a9d3b2c4d8b00001b","seq":1,"position":10,"parentId":"4166d6f5dd5490308d000033","content":"### Interface [Module]\n\nA concept module specifically for the description of plugin modules."},{"_id":"41674cc7dd5490308d00003f","treeId":"4130f67a9d3b2c4d8b00001b","seq":1,"position":11,"parentId":"4166d6f5dd5490308d000033","content":"### Remote Plugin [Module]\n\nA plugin module which is dynamically loaded over a network, interprocess, or similar connection (i.e. the plugin is actually running on another machine)."},{"_id":"41674eb5dd5490308d000040","treeId":"4130f67a9d3b2c4d8b00001b","seq":1,"position":12,"parentId":"4166d6f5dd5490308d000033","content":"### Remote Bridge [Module]\n\nA bridge module which wraps a live service (like an API)."},{"_id":"4172fe14b90f0cca36000086","treeId":"4130f67a9d3b2c4d8b00001b","seq":1,"position":1.25,"parentId":"4166d509dd5490308d000032","content":"## World/Noun/Verb\nMarkers within the module system which keep track of objects and actions, and how they relate to each other.\n\nSpecifically the entity system for the modules keeps track of the interesting objects related to each module, and other objects register themselves as interesting in other ways, each registration contains a description object which allows the objects to be organized, typed, and compared.\n\nVerbs are also added to the system, whether as compound actions which invoke other verbs, or as singular verbs. These verbs can provide different capabilities and information. For example descriptions of the types of objects they require, alternate functions which can pretend to do the action if given mocked up object types, etc. Others may out right describe the action they perform.\n\nVerbs always take a world, every noun is part of a world. Many nouns provide functionality for creating mocked versions in a 'fake' world. World use contexts.\n\nThis system is used by the AI to perform actions across modules. And by the user interface for generating interactive consoles and completing input. "},{"_id":"41675d19dd5490308d000043","treeId":"4130f67a9d3b2c4d8b00001b","seq":1,"position":1.5,"parentId":"4166d509dd5490308d000032","content":"## Description\n##### Module\n\nEvery module has a description. This description includes information about the purpose of the module, how to use it, and how it should affect other parts of the system. The description is what drives the majority of the other features of the module system.\n\nEach description has an arbitrary number of sub parts, depending on the features available for parsing it. Each description component has a list of dependencies from which it comes from. Descriptions are acquired in various stages.\n\nIt's important to note that types of information generated, and the formats containing information are disjoint. One file could contain identity, forge, and code descriptions. Description information can also be inherited from libraries (calculated at reduce time)."},{"_id":"416761b6dd5490308d000044","treeId":"4130f67a9d3b2c4d8b00001b","seq":1,"position":1,"parentId":"41675d19dd5490308d000043","content":"### Identity Description\n\nThe description of the identity of the plugin. Namely for versioning, namespacing, organizing, and searching. But probably most importantly for uniqueness and security. The identity description includes signature descriptions, and all of the identity domain descriptions can be used for uniqueness. Some example components within the identity description domain:\n\n* Custom Name(s)\n* Author(s)\n* (Cryptographic) Signatures.\n* Guids.\n* Organizational Locations (AKA, Namespaces).\n* Hash.\n* Previous Version:\n * Signature.\n * Hash.\n * Guid."},{"_id":"4167b441dd5490308d000048","treeId":"4130f67a9d3b2c4d8b00001b","seq":1,"position":1.5,"parentId":"41675d19dd5490308d000043","content":"### Dependency Description\n\nA description of the other modules this one relies on. Different modules are relied on in different ways, and those relationships may describe how to build other modules based off of the relationship."},{"_id":"41679cf8dd5490308d000046","treeId":"4130f67a9d3b2c4d8b00001b","seq":1,"position":2,"parentId":"41675d19dd5490308d000043","content":"### Code Description\n\nThis is a description of code, often including the code within this module, but could describe code in other modules. Often relies on identity to provide unique locations of some types within it.\n\nNamely this module is responsible for describing the way code modules interact, for example:\n\n* Types:\n * Abstract.\n * Concrete, in this module.\n * Object System Types.\n * Generic/Parametric Types.\n * Other System Types.\n * Type Relationships.\n* Variables, in this module.\n* Operators, across multiple modules."},{"_id":"4167a106dd5490308d000047","treeId":"4130f67a9d3b2c4d8b00001b","seq":1,"position":3,"parentId":"41675d19dd5490308d000043","content":"### Forge Description\n\nThis is a description of how to build, wrap, load, and execute a code module. Relies on the code description and dependency description to do some operations (but not others). Namely contains:\n\n* Inter-module Protocol(s) used for which plugin dependencies.\n* Libraries(s) used.\n* Language(s) used. Which computes/contains:\n * Compiler(s) used, and it's targets.\n * Linker(s) used, and it's targets.\n * Interpreter(s) used, and it's targets.\n * Runtime(s) used.\n * Wrapper features/providers used (also depends on the inter-module protocols and libraries)."},{"_id":"41675aeadd5490308d000042","treeId":"4130f67a9d3b2c4d8b00001b","seq":1,"position":2,"parentId":"4166d509dd5490308d000032","content":"## Identity\n##### Module\n\nPerhaps the most important part of a module is it's identity. However identities exist for many other things. For example, both forges (and their components) and code symbols have identities.\n\nAn identity provides a couple of key features:\n* Searching: Identities provide a way to search for an instance of something, based on any of it's properties.\n* Namespacing: Identities provide organizational information which lets them be organized into ontologies of instances.\n* Versioning: Identities provides the ability to differentiate between different versions of the same thing. But also for choosing and requesting a specific version of something.\n* Security: Identities provide security that the plugin you are using is safe to use, based off of the trusted authorities for each plugin group.\n* Uniqueness: Identities are unique, and provide ways of matching range identities to unique instances.\n\nWhen used for code symbols, for example, the identity system is used to find the symbol in question, and other partial matches.\n\nWhen used for forges, for example, the identity system is used for searching, versioning, and compatibility.\n\nFinally, when used for module, for example, the identity system provides searching and matching."},{"_id":"4172e27fb90f0cca36000084","treeId":"4130f67a9d3b2c4d8b00001b","seq":1,"position":1,"parentId":"41675aeadd5490308d000042","content":"### Identity\n\nA singular identity is a collection of information describing a unique entity. Some of this information may be indexed, or otherwise used to provide features, but an identity is singular."},{"_id":"4172e522b90f0cca36000085","treeId":"4130f67a9d3b2c4d8b00001b","seq":1,"position":2,"parentId":"41675aeadd5490308d000042","content":"### Identity Range\n\nAn identity range describes a potential range of identities, and is attempting to find a single one. Often there are resolution methods attached to the range which can choose the best choice among a list of plugins."},{"_id":"4172d721b90f0cca36000082","treeId":"4130f67a9d3b2c4d8b00001b","seq":1,"position":2.25,"parentId":"4166d509dd5490308d000032","content":"## Projection\n\nA projection is an operation performed on information (like a description or source code) which produces another view, or format, of the source. Typically projections are one way, although they don't need to be. For our purposes projections means an operation which takes information in an in memory machine format (for example, in a parse tree, or the generated semantic tree) and produces another format meant for storage or reading."},{"_id":"4172d334b90f0cca36000081","treeId":"4130f67a9d3b2c4d8b00001b","seq":1,"position":2.5,"parentId":"4166d509dd5490308d000032","content":"## Artifact\n\nAn artifact is the projection of some of the description and source code of a plugin, from which the description and source is likely not recoverable, and more specifically, which is designed to be executable. That is an executable projection."},{"_id":"4172e13bb90f0cca36000083","treeId":"4130f67a9d3b2c4d8b00001b","seq":1,"position":1,"parentId":"4172d334b90f0cca36000081","content":"### Artifact Target\n\nEvery artifact has a target, which is an identity range of what it needs to be useable on. The matching identity is the target actually being ran on. The target includes information like:\n\n* Dll Identity Ranges. Including version, name, security information. Often translated into an OS specific format as part of the artifact.\n* The hardware required, for example SSE extensions, and bit width."},{"_id":"4172cf38b90f0cca36000080","treeId":"4130f67a9d3b2c4d8b00001b","seq":1,"position":3,"parentId":"4166d509dd5490308d000032","content":"## Forge\n\nForges are a collection of code which is responsible for generating artifacts for code modules. A forge may be responsible for multiple code modules, and multiple types of artifacts.\n\nForges contain a lot of parts, specifically:\n\n* Wrapper Generator Fragments.\n* Compiler Configurators.\n* Linker Configurators.\n* Compilation Managers.\n* Protocol Endpoint Configurators.\n* Library Configurators.\n* Runtime Configurators.\n* Interpreter Configurators.\n\nGenerally the design of the forge is that some parts provide new Actions for the plugins whose forge identities match a forge range. These actions are often limited to the target being attempted."},{"_id":"41730e17b90f0cca36000087","treeId":"4130f67a9d3b2c4d8b00001b","seq":1,"position":4,"parentId":"4166d509dd5490308d000032","content":"## Reports\n\nReports represent another entity component system. Specifically a way of describing the result of some action. Reports are split into trees of results of the action. And provide human readable data. And potentially internal madz references to other reports or available features.\n\nThey are meant to provide a dynamic way for people to inspect the results of madz actions, and view the current state."},{"_id":"41cd0bb9e3d3cbe18700004e","treeId":"4130f67a9d3b2c4d8b00001b","seq":1,"position":1,"parentId":"41730e17b90f0cca36000087","content":"### Action Reports\n\nNearly every piece of code which runs in madz is tied to an action, some driving force which caused it to be executed. Reports are hence organized primarily by the action that they were responsible for. This is the entity used in reports, the action that caused the code to be ran."},{"_id":"41cd1279e3d3cbe18700004f","treeId":"4130f67a9d3b2c4d8b00001b","seq":1,"position":2,"parentId":"41730e17b90f0cca36000087","content":"### State Reports\n\nState Reports express the current state of the madz system, including the latest action reports in each area. For example every time a plugin is built, it's latest action report is used as the current state of that plugin."},{"_id":"41cd15aee3d3cbe187000050","treeId":"4130f67a9d3b2c4d8b00001b","seq":1,"position":3,"parentId":"41730e17b90f0cca36000087","content":"### Object Reports\n\nA subset of state reports, these are collections of the latest action reports for a specific object."},{"_id":"41cd17abe3d3cbe187000051","treeId":"4130f67a9d3b2c4d8b00001b","seq":1,"position":4,"parentId":"41730e17b90f0cca36000087","content":"### Dynamic Reports\n\nReports have the capability to be dynamic, to be updated when information changes, to provide the most up to date state of the system. And also to provide links to objects, and their actions."},{"_id":"41cd1b3ee3d3cbe187000052","treeId":"4130f67a9d3b2c4d8b00001b","seq":1,"position":5,"parentId":"41730e17b90f0cca36000087","content":"### Report Logs\n\nPerhaps the most commonly used part of reports are the logs they contain, which contain text, and objects, about what happened. Report logs contain tags, and other features, for organizing them."},{"_id":"4131ce0f9d3b2c4d8b000027","treeId":"4130f67a9d3b2c4d8b00001b","seq":1,"position":2,"parentId":null,"content":"# Vision\n\nSome examples and descriptions of what madz should provide. Vision is split into different categories of what madz will provide.\n\nThe core point of madz is to manage software projects. This ranges from **checking out** the source code, **exploring** (aka 'reading') the source code, **organizing** the source code, **adding** to the source code, **editing** the source code, **committing** the source code, **building** the source code, **testing** the source code, **deploying** the source code, **debugging** the source code, **reproducing errors** with the source code, **documenting** the source code, **importing** external source code, and **exporting** existing source code and any other source code manipulation.\n\nIt also provides features for **managing libraries, SDKs and APIs**, and **custom tool chains** used by the developer."},{"_id":"4131ceba9d3b2c4d8b000028","treeId":"4130f67a9d3b2c4d8b00001b","seq":1,"position":3,"parentId":null,"content":"# Design\n\nA highlevel discussion of feature design, at multiple levels of madz. Design is split into abstraction layers, each of which represents a set of abstractions used by components and systems of madz at that layer, as well as the implementation of features at that layer.\n\nSome of the abstraction layers described here will in reality be separate libraries originally designed for madz."},{"_id":"4616357eb10ec6a40100008f","treeId":"4130f67a9d3b2c4d8b00001b","seq":318629,"position":0.5,"parentId":"4131ceba9d3b2c4d8b000028","content":"## Top-Level Organization\n\nMadz is not a program, and it is not an application, at least not in the traditional sense. Madz is a library for managing data, with many high level objects cooperating to deliver powerful features.\n\nGenerally madz will have a toplevel set of bindings which represent the entire context of the current madz execution. These are global variables (actually thread local to allow for concurrent execution) which provide the top level objects of madz which orchestrate it's features.\n\nThese toplevel bindings may be manipulated to provide different environments as required.\n\nThe list of top level objects:\n* Module Ecs (Bootstrap Ecs)\n* File Ecs\n* Report Ecs\n* Taskmaster\n* Forge Architect\n* Planner"},{"_id":"46164288b10ec6a401000090","treeId":"4130f67a9d3b2c4d8b00001b","seq":318608,"position":1,"parentId":"4616357eb10ec6a40100008f","content":"### Entity Component Systems\n\nEntity component systems represent the live data stores for many of the more dynamic entity types, or more precisely, for ontological (as opposed to taxonomical) based object systems. Primarily modules (and the bootstrapped modules), although reports and files are also setup as entity component systems.\n\nThese systems are designed to provide objects which are ontological in nature, and provide support structures for those objects to act like any object required."},{"_id":"41ccf57de3d3cbe18700004d","treeId":"4130f67a9d3b2c4d8b00001b","seq":318524,"position":1,"parentId":"4131ceba9d3b2c4d8b000028","content":"## Description Loading\n\nDescription loading is a lazy, resume-able, multistage process to find descriptions of modules.\n\n* All registered description acquirers are executed across the format. These are arranged in a dependency tree, and acquire all of the possible information from a module. Often via inspection, unpacking, etc. (For example, file based acquirers will list the file listing description as a dependency). Once an acquirer has determined a module isn't the primary one in question, it will pause until the module's description is required (this will allow lazy networked module acquirers).\n* All registered description expanders are executed across the 'base unexpanded description' these expanders are meant to expand and reformat the existing information into other usable formats. In many cases, these formats will be lazy (being added, but only actually expanded if asked for). Also arranged in a dependency tree.\n* Once all modules related to the primary one have a 'base expanded description', all registered description reducers synthesize global objects containing indexes, and combined information about descriptions, additionally base descriptions may be updated to contain references into the synthesized objects (this is domain dependent). It's at this point that inconsistencies are detected, and hopefully, resolved. Also executed in a dependency tree."},{"_id":"41d285afe906518443000053","treeId":"4130f67a9d3b2c4d8b00001b","seq":1,"position":1,"parentId":"41ccf57de3d3cbe18700004d","content":"### Description Detectors\n\nDetect new module descriptions."}],"tree":{"_id":"4130f67a9d3b2c4d8b00001b","name":"MAssive DangerZone Reference Document","publicUrl":"massive-dangerzone"}}