Full explainability provenance for all patterns with new RDF types (Decomposition, Finding, Plan, StepResult, Synthesis) and predicates (tg:subagentGoal, tg:planStep)
Analysis split into Analysis+ToolUse and Observation for finer-grained DAG provenance; message_id wired on all streaming answer chunks
CLI support for pattern selection: tg-invoke-agent -p supervisor|plan-then-execute|react
RabbitMQ Pub/Sub Backend (#751, #752, #765): Pub/sub abstraction decoupled from Pulsar with RabbitMQ as an alternative backend, demonstrating independence from any single messaging fabric. RabbitMQ was selected for its significantly lower resource requirements compared to Pulsar. Support for additional fabrics such as Kafka is planned for a subsequent release.
Selectable via PUBSUB_BACKEND=rabbitmq environment variable
Topic exchange architecture with shared and exclusive consumer queues
Translator rename: to_pulsar/from_pulsar → encode/decode across 55+ files
Queue naming format changed to CLASS:TOPICSPACE:TOPIC
Subscriber resilience: automatic consumer recreation after connection failure
Thread-safe consumer model with dedicated thread pools for pika
SPARQL Query Service (#754, #755): Backend-agnostic SPARQL 1.1 query service:
Parses SPARQL queries using rdflib, decomposes into triple pattern lookups via existing pub/sub interface
Supports BGP, JOIN, OPTIONAL, UNION, FILTER, BIND, VALUES, GROUP BY, ORDER BY, LIMIT/OFFSET, DISTINCT, and aggregates
Batching and streaming support for large result sets
Gateway integration, Python SDK method (FlowInstance.sparql_query), and CLI command (tg-invoke-sparql-query)
Universal Document Decoder (#705): Multi-format document processing using the unstructured library:
Supports DOCX, XLSX, PPTX, HTML, Markdown, CSV, RTF, ODT, EPUB and more through a single service
Tables preserved as HTML markup; images stored in librarian
All decoders now share the document-decoder ident for interchangeability
Improvements
Inline Explainability Triples (#763): Provenance triples now included directly in explain messages from GraphRAG, DocumentRAG, and Agent services, eliminating follow-up knowledge graph queries for explainability details
Config Push Notify Pattern (#760): Replaced stateful pub/sub config broadcast with lightweight notify signal containing only version number and affected config types
Persistent WebSocket Connections (#723): Single persistent connection with request multiplexing replaces per-request WebSocket connections, eliminating repeated TCP+WS handshakes. CLI tools converted to concurrent WebSocket requests
Auto-pull Ollama Models (#757): Ollama provider automatically pulls missing models on first use
MCP Gateway Auth (#721): GATEWAY_SECRET environment variable support for MCP server to API gateway authentication
Chunk Content ID in Explain Traces (#708): tg-show-explain-trace now displays chunk URIs with --show-provenance for easy source text retrieval via tg-get-document-content
Prompt Queue Monitoring (#737): New tg-monitor-prompts CLI tool for subscribing to prompt request/response queues with correlation and timing summaries
Bug Fixes
Dispatcher Race Condition (#715): Fixed duplicate dispatcher creation under concurrent coroutines causing dropped responses and permanent UI spinners
WebSocket Error Responses (#726): Fixed missing request IDs in websocket multiplexer error responses causing client hangs on failed requests
OpenAI Compatibility (#727): Use max_completion_tokens instead of deprecated max_tokens for newer OpenAI/Azure models; added AZURE_API_VERSION environment variable override
Missing Auth Header (#724): Fixed verify_system_status processor check not including authorization header when gateway auth is enabled
Gateway Text Load (#729): Accept raw UTF-8 text in text-load endpoint
Stray Log Messages (#706): Removed spurious warnings from librarian responses arriving on shared response queues
Consumer Poll Timeout: Reduced consumer poll timeout from 2000ms to 100ms for improved responsiveness
Breaking Changes
Pub/sub queue naming: Queue format changed from topic-based to CLASS:TOPICSPACE:TOPIC; translator methods renamed from to_pulsar/from_pulsar to encode/decode
Config push schema: ConfigPush now contains a types list instead of the full config dict; state queue class replaced by flow class
Infrastructure / Technical
Testing (#745, #749, #750): 96+ orchestrator tests covering aggregation, provenance, routing, explainability parsing, DAG structure, and callback message IDs
CLA Workflow (#716, #722): Contributor License Agreement process via GitHub action
Pulsar Check Skipped (#753): tg-verify-system-status no longer requires Pulsar when using alternative pub/sub backends
v2.1 (2026-03-17)
Major Features
Explainability & Provenance (#655, #661, #677, #682, #688, #689, #693, #694, #697, #698): End-to-end explainability across the entire pipeline:
Extract-time provenance: Document processing now emits PROV-O triples tracing the lineage from documents through pages, chunks, and extracted edges using prov:wasDerivedFrom relationships
Query-time explainability: GraphRAG, DocumentRAG, and Agent queries record full reasoning traces (question, grounding, exploration, focus, synthesis stages) into a dedicated urn:graph:retrieval named graph
Named graphs: Knowledge is now stored across named graphs — default graph for facts, urn:graph:source for extraction provenance, urn:graph:retrieval for query-time explainability
Subgraph provenance: Extracted subgraphs are tracked with provenance linking edges back to their source chunks and documents
New CLI tools: tg-list-explain-traces, tg-show-explain-trace, tg-show-extraction-provenance
Explainability modes added to tg-invoke-graph-rag, tg-invoke-document-rag, and tg-invoke-agent with inline provenance event display
Value to Term Schema Redesign (#622): Breaking redesign of the core wire format from Value ({"v": ..., "e": true}) to typed Term format:
Tool service client for the ReAct agent to discover and invoke tools at runtime
Tools can be deployed independently and registered dynamically
Batch Embeddings (#668, #669, #670, #671, #672, #681): Embeddings service redesigned for batch processing:
embed() now accepts a list of texts instead of a single text
Updated all embeddings providers (FastEmbed, Ollama, etc.)
Embeddings API now returns similarity scores
New CLI tools: tg-invoke-embeddings, tg-invoke-graph-embeddings, tg-invoke-document-embeddings, tg-invoke-row-embeddings
Improvements
Incremental / Large Document Loading (#659, #660): Multipart upload support for large documents:
S3 multipart upload with streaming retrieval
Upload session tracking in Cassandra with 24-hour TTL
New REST endpoint GET /api/v1/document-stream for streaming document content
New CLI tool: tg-get-document-content
Entity-Centric Graph (#633): Redesigned graph schema for entity-centric storage and querying
Structured Data Enhancements (#645, #646): Multi-index table support for structured data, removing need for manual Cassandra table modifications:
Row embeddings APIs exposed through gateway
New row-embeddings-query tool type for semantic search on structured data indexes
Streaming Triples (#676): Streaming triple queries with configurable batch sizes for lower time-to-first-result and reduced memory overhead:
tg-show-graph updated with --limit, --batch-size, --graph filter, and --show-graph options
Graph Query CLI (#679): New tg-query-graph tool for selective pattern matching on the triple store (by subject, predicate, object, graph) with auto-detection of value types
RDF-star Support in Turtle Export (#676): tg-graph-to-turtle now handles quoted triples and named graph filtering
Enhanced GraphRAG Pipeline (#691, #697): 4-stage GraphRAG pipeline with query concurrency and DocumentRAG grounding
Prompts JSONL Format (#619): Support for JSONL format in prompt definitions
Entity Context Enhancement (#629): Entity term now output alongside its definition in entity contexts
Terminology Rename (#682): Clarified naming throughout — “provenance” callbacks/IDs renamed to “explain” for clarity
Bug Fixes
Cassandra Schema and Graph Filter Semantics (#680): Fixed Cassandra schema for named graph support and corrected graph filter semantics
Testing: Added comprehensive Python API client tests with streaming validation
v1.6 (2025-12-04)
Major Features
Streaming LLM Responses (#566, #567): Comprehensive streaming support for LLM text completion enabling real-time token-by-token delivery:
Infrastructure with streaming flag in schemas, Gateway API (REST/WebSocket), Python API, and CLI tools.
Full streaming implementation across all LLM providers including Azure, Azure OpenAI, Bedrock, Claude, Cohere, Google AI Studio, Llamafile, LM Studio, Mistral, Ollama, OpenAI, TGI, Vertex AI, and vLLM
Backward compatible with existing non-streaming clients
Support for WebSocket streaming
Reduces time-to-first-token and improved UX for long responses
Streaming RAG Responses (#568): Extended streaming support to GraphRAG and DocumentRAG services:
Token-by-token responses for knowledge graph and document retrieval queries
Consistent streaming UX across all TrustGraph services
Gateway support via WebSocket for real-time client applications
Streaming Agent Interactions (#570): Enhanced agent framework with streaming support:
Real-time streaming of ReAct agent thought/observation/answer chunks
Incremental response delivery for multi-step agent workflows
Streaming parser for agent responses with robust error handling
Improvements
Enhanced Integration Tests (#568, #570): Comprehensive test coverage for streaming functionality
Bug Fixes
AWS Bedrock Model Invocation (#572): Fixed compatibility issues with newer Bedrock model invocation API including proper streaming support
Minio Library Compatibility (#565): Fixed incompatible library change in Minio client for blob storage operations
Streaming Agent Interactions (#570): Fixed race conditions and message ordering issues in streaming agent responses
Infrastructure / Technical
CLI Improvements: Enhanced CLI tools with streaming output:
tg-dump-queues: New utility for developer queue diagnostics
Templates
Updates to Bedrock and Claude models to support latest models
v1.5 (2025-11-23)
New Features
OntoRAG: Ontology-Based Knowledge Extraction (#523): New processor kg-extract-ontology that uses ontology objects from config to guide triple extraction. Includes entity contexts and integrates with ontology extractor from workbench.
MCP Authentication (#557): Added MCP auth token header support for the simple authentication case.
Dynamic Embeddings Model Selection (#556): Embeddings model can now be selected dynamically rather than being fixed.
Bug Fixes
Collection deletion batch error (#559): Fixed batch error during collection deletion by reducing batch size.
Fix hard-coded vector size (#555): Vector store now lazy-creates collections with different collections for different dimension lengths.
Fix AgentStep schema error (#557): Agent step argument values are now converted to strings to fix schema errors.
Remove unnecessary OpenAI parameters (#561): Removed parameters from OpenAI invocation that were causing compatibility issues with OpenAI-compatible services.
Infrastructure / Technical
Python 3.13 Support (#553): Upgraded to Python 3.13, switched from cassandra-driver to scylla-driver (cassandra-driver doesn’t work with Python 3.13).
Vector Store Lifecycle Tech Spec (#555): Added technical specification for vector store lifecycle.
OntoRAG Tech Spec (#523, #558): Added technical specification for ontology-based knowledge extraction and query.
Testing
Ontology extraction tests (#560): Added tests for ontology extraction.
Dynamic embeddings tests (#556): Added tests for dynamic embeddings model selection.
MCP auth and agent step parsing tests (#557): Added tests for MCP authentication and agent step parsing.
OpenAI invocation tests (#561): Updated tests for OpenAI parameter changes.
v1.4 (2025-10-06)
New Features
Flow Configurable Parameters (#526, #530, #531, #532, #533, #541): Major enhancements to flow parameter system:
Flow configurable parameters with dynamic settings
LLM dynamic settings using llm-model and llm-rag-model parameters
Temperature parameter support for all LLMs
Flow creation uses parameter defaults in API and CLI
Advanced parameter mode with controlled-by relationships
New CLI tools: tg-show-parameter-types
Dynamic chunking parameters
Structured Data Diagnosis Service (#518, #519):
New structured data diagnosis service plumbed into API gateway